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Objective 

The  overall  objective  of  this  project  was  to  design,  develop. 
Implement,  and  evaluate  an  authoring  system  which  would  provide  a 
basis  for  cost  effective  production  of  computer-assisted  Instruction 
(CA1)  materials  for  use  in  the  context  of  computer-managed  Air  Force 
technical  training.  The  specific  target  application  was  the  Advanced 
Instructional  System  (AIS)  located  at  Lowry  Air  Force  Base,  Colorado, 
and  the  software  developed  was  to  be  Integrated  into  this  system.  The 
project  work  was  conducted  through  two  parallel  efforts  which  are 
described  in  Parts  I  and  II  of  this  two-part  report.  The  first  of  these 
efforts,  described  here  In  Part  I  of  the  report,  addressed  the  develop¬ 
ment  of  computer  software  to  facilitate  the  authoring,  presentation, 
and  evaluation  of  CAI  materials.  The  second  effort,  described  in  Part 
II,  concerned  the  definition  of  a  procedural  model  for  CAI  production 
and  evaluation  of  the  complete  authoring  system. 

Approach 

The  design  activities  began  with  analysis  of  the  probable  functions 
of  CAI  within  the  AIS,  review  of  prior  approaches  to  supporting  CAI 
materials  development,  examination  of  previous  military  CAI  development 
experiences,  analysis  of  the  characteristics  (training,  prior  experience 
and  work  environment)  of  the  Air  Training  Comnand  (ATC)  personnel  who 
would  be  developing  CAI  materials,  and  re-examination  of  the  available 
AIS  software  and  operational  experience  with  this  software.  A  major  con¬ 
clusion  resulting  from  these  analyses  was  that  there  are  a  number  of 
factors  in  the  military  technical  training  environment  which  are  incom¬ 
patible  with  the  typical  approach  to  CAI  production  (authoring)  and  that 
Drior  attempts  to  utilize  CAI  in  this  environment  had  not  taken  these 
factors  into  sufficient  consideration.  Typically,  it  appeared  that  too 
much  was  expected  of  the  CAI  authors.  It  was  decided,  therefore,  that 
it  was  preferable  to  adapt  the  authoring  system  to  the  existing  environ¬ 
ment  rather  than  expect  the  environment  to  change  to  meet  the  require¬ 
ments  of  the  system,  even  when  this  approach  limited  the  sophistication 
of  the  CAI  materials  which  could  be  produced.  For  example,  it  was  con¬ 
sidered  preferable  to  avoid  author  use  of  a  programming  language  even 
though  this  would  limit  the  author's  flexibility.  The  AIS  employs 
editors  which  engage  the  user  in  an  interactive,  English  language  dialog 
to  control  the  system's  data  base.  Experience  with  these  interactive 
editors  suggested  that  they  could  provide  a  model  for  the  author/computer 
interface.  It  was  also  concluded  that  the  system  should  structure  the 
author's  task,  promote  principles  of  good  instructional  design,  require 
very  little  training  for  its  use,  and  provide  aids  for  managing  a 
materials  development  project. 

The  heart  of  the  software  developed  is  an  interactive  CAI  Authoring 
Editor.  The  Editor  structures  the  author's  task  while  providing  options 
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as  to  CAI  presentation  strategy  details.  As  defined  by  the  Lditor,  a 
CAI  module  is  divided  into  objectives.  The  system  is  frame  oriented  and 
each  objective  can  contain  up  to  1 00  frames.  Three  classes  of  frame 
types  are  supported:  textual  content  frames;  question  frames  and  special 
purpose  frames.  The  author  enters  the  frame  content  in  exactly  the 
format  in  which  it  will  be  seen  by  the  student.  All  formatting  of 
question  frames  is  done  automatically  and  the  autnor  is  prompted  to 
supply  a  feedback  statement  for  each  alternative  or  constructed  response 
and  a  prompt  statement  for  each  attempt  at  answering  the  question.  To 
individualize  instruction,  the  author  can  define  branches  from  any  frame 
to  any  other  frame.  The  decision  of  whether  or  not  to  branch  can  be 
based  on  a  specified  number  of  questions  being  answered  correctly  or 
incorrectly,  on  a  set  of  frames  having  or  not  having  been  presented,  or 
on  a  specific  student  response.  Branching  logic  is  entered  in  a  highly 
prompted,  multiple  choice  format,  and  the  result  is  then  displayed  in 
English. 

CAI  materials  are  delivered  by  a  CAI  Presentation  Program.  A 
standard  presentation  program  was  first  written  for  CAI  as  an  alternative 
instructional  module.  Two  modified  versions  of  this  program  provide 
review  and  remeoiation  over  the  specific  objectives  with  which  the 
stuuent  had  problems.  The  presentation  programs  contain  student  per¬ 
formance  data  collection  routines  which  can  be  turned  on  or  off  by  the 
author.  Standard  data  analysis  reports  are  requested  by  means  of  a 
second  interactive  editor  which  prompts  the  user  as  to  the  options  avail¬ 
able. 

A  variety  of  printouts  of  program  content  and  logic  are  provided. 
Module  content  can  also  be  printed  out  in  a  format  suitable  for  student 
use  as  a  programed  text. 

Evaluation  Procedures  and  Results 

The  software  developed  was  evaluated  as  part  of  the  total  authoring 
system  through  the  production  and  implementation  of  a  set  of  CAI  lessons 
in  one  of  the  courses  supported  by  AIS  and  by  training  a  number  of  course 
personnel  in  the  use  of  the  authoring  system.  The  evaluation  procedures 
and  results  are  described  in  detail  in  Part  II  of  this  report  and  are 
only  summarized  here,  in  Part  I. 

Under  the  materials  production  effort,  CAI  modules  were  developed 
for  six  lessons  in  the  AIS  Weapons  Mechanic  course.  None  of  the  three 
members  of  the  authoring  team  had  prior  CAI  experience;  although,  all 
were  experienced  technical  training  authors.  Approximately  CC00  man 
hours  were  required  for  development  of  the  six  modules.  The  modules 
accounted  for  a  total  of  approximately  2b  Plan  of  Instruction  (POI) 
hours  and  resulted  in  an  average  student  contact  time  of  18.7  hours. 

Thus,  development  efforts  required  an  average  of  88  man  hours  per  POI 
hour  and  118  man  hours  per  student  contact  hour.  This  compares  very 
favorably  with  the  figures  of  222  and  246  man  hours  per  contact  hour 
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reported  by  Himwich  (1977)  for  military  technical  training  CAI. 

Three  ATC  instructors  were  trained  in  use  of  the  Authoring  Editor 
during  15  one-half  day  sessions.  None  of  the  trainees  were  computer 
progranmers  or  had  any  prior  CAI  development  experience.  There  was  no 
formal  training  after  the  first  session.  Given  a  CAI  Author's  Handbook 
to  use  as  a  reference  manual,  each  trainee  used  the  Authoring  Editor  to 
develop  a  CAI  module.  Contractor  personnel  were  available  to  answer 
questions  and  review  the  trainees'  work.  At  the  end  of  the  training 
period,  each  had  developed  a  module  through  the  stage  of  revision  follow¬ 
ing  single  student  tryouts.  The  trainees  asked  relatively  few  questions 
and  the  modules  produced  were  of  generally  good  quality  and  capitalized 
fairly  well  on  the  capabilities  of  CAI.  The  trainees  were  quite 
satisfied  with  the  Editor  and  all  expressed  an  interest  in  implementing 
CAI  in  their  courses. 

Conclusions 


The  approach  taken  to  facilitating  CAI  development  appears  very 
promising.  Experience  to  date,  while  limited,  has  demonstrated  that  con¬ 
tractor  personnel  were  able  to  produce  effective  CAI  with  a  level  of 
effort  that  is  comparable  to  the  effort  required  to  produce  paper  and 
pencil  materials.  ATC  personnel  were  able  to  learn  to  use  the  Authoring 
Editor  and  to  produce  CAI  materials  within  a  very  short  period  of  time. 
These  trainees  expressed  highly  favorable  attitudes  about  the  approach 
and  found  no  serious  faults  with  the  Editor. 

The  major  factors  which  contribute  to  simplifying  the  authoring 
task  are  probably  elimination  of  any  need  for  the  author  to  use  a  com¬ 
puter  language  and  the  extent  to  which  the  task  is  structured.  The  human 
engineered,  computer-aided  input,  formatting,  and  editing  capabilities 
provided  by  the  Editor  are  undoubtedly  also  important. 

The  authoring  system  is  ready  for  use  by  ATC  instructional  develop¬ 
ment  personnel  in  its  current  form  but  it  should  not  necessarily  be 
considereu  to  be  a  finished  product.  There  are  a  number  of  features 
which  could  be  added  to  increase  its  utility.  As  the  system  is  used,  it 
can  be  expanded  and  refined  on  the  basis  of  accumulating  experience  to 
become  a  powerful  tool  for  instructional  development. 
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I  INTRODUCTION 


Computer-based  instruction  has  the  potential  of  achieving  signif¬ 
icant  savings  in  costs  related  to  Air  Force  technical  training.  A  number 
of  steps  toward  realizing  this  potential  have  already  been  taken,  e.g., 
use  of  the  PLATO  system  and  development  of  the  Advanced  Instructional 
System  (AIS).  The  former  is  an  application  of  computer-assisted  in¬ 
struction  ( CA I )  while  the  latter  represents  a  comprehensive  computer- 
based  instructional  (CBI)  system  supporting  both  computer-managed  in¬ 
struction  (CMI)  and  CAI. 

CMI  can  be  defined  as  a  situation  in  which  the  majority  of  the 
stuoent's  instructional  activities  are  completed  off-line.  The 
computer's  role  is  that  of  evaluator,  diaqnostician,  prescriber,  and 
manager  of  instructional  events.  In  CAI,  by  contrast,  all  of  the  stu¬ 
dent's  instructional  activities  are  conducted  on-line,  at  an  inter¬ 
active  computer  terminal.  CMI  can  be  characterized  as  being  extensive, 
managinq  instruction  for  a  large  number  of  students  throughout  a  large 
body  of  course  content.  CAI,  on  the  other  hand  is  typically  intensive, 
concentrating  on  detailed  and  hiqhly  interactive  instruction  for  a 
limited  segment  of  course  content  and  a  relatively  small  number  of 
students,  extensive  application  of  CAI  has,  to  date,  been  limited  by  its 
high  cost,  in  terms  of  both  materials  production  and  terminal  costs, 
and  by  the  limited  utility  of  insular  segments  of  individualized  instruc¬ 
tion  in  a  group-paced  environment. 

The  work  reported  herein  addresses  the  concept  of  a  comprehensive 
instructional  system  in  which  CAI  is  embedded  in  the  context  of  CMI. 

Such  an  integrated  system  has  a  number  of  distinct  advantages.  Student 
pacing,  being  individualized,  is  compatible  with  efficient  use  of  CAI. 
Student  performance  on  CAI  lessons  can  be  recorded  directly  by  the  CMI 
system.  Most  importantly,  the  extensive  student  performance  records 
maintained  as  part  of  CMI  can  be  readily  accessed  to  provide  truly 
individualized  CAI,  when  and  where  it  is  most  needed. 

The  relatively  expensive  CAI  must,  however,  be  employed  judiciously 
in  the  CMI  context  to  be  cost  effective.  If  one  assumes  that  the  off¬ 
line  instructional  materials  being  managed  by  CMI  are  reasonably 
effective,  the  use  of  CAI  is  only  appropriate  when  (a)  the  concepts  or 
facts  to  be  taught  are  uniquely  difficult  and  existing  instructional 
materials  and  methods  are  Inadequate  for  a  large  proportion  of  the 
students,  or  (b)  logistical  problems  can  be  resolved  through  the  use  of 
CAI.  The  cost  of  producing  and  delivering  CAI  for  specific  applications 
is,  of  course,  also  an  important  consideration. 

Project  Purpose 


There  are  two  basic  reasons  why  CAI  production  costs  remain  high. 
With  few  exceptions,  the  programming  languages  and  production  methods 


employed  require  extensive  and  very  detailed  effort.  Secondly, 
relatively  sophisticated  instructional  design  skills  are  required  as 
well  as,  in  most  cases,  computer  programing  skills. 

A  basic  premise  of  this  project  was  that  in  the  military  technical 
training  environment,  CAI  development  problems  can  be  alleviated  and 
production  costs  reduced  through  structuring  the  authoring  task  and  pro¬ 
viding  software  tools  to  facilitate  the  authoring  process.  The  term 
"authoring,"  as  it  is  used  in  this  report,  refers  to  the  process  of 
generating,  evaluating,  and  revising  an  individualized.  Interactive  CAI 
module.  Such  a  module  consists  of  information  to  be  assimilated  by  the 
student,  questions  and/or  practice  exercises,  decision  rules  for 
individualizing  the  amount  and  nature  of  the  instruction  on  the  basis  of 
student  performance,  and  questions  or  exercises  to  measure  the  student's 
mastery  of  the  module's  objectives.  It  was  hypothesized  that  appropriate 
software,  designed  specifically  to  support  CAI  authoring  in  this  environ¬ 
ment,  could  directly  and  substantially  reduce  the  cost  of  CAI  development 
and  evaluation.  It  was  also  thought  that  skill  level  requirements  and 
production  time  could  be  further  reduced  if  the  design,  development, 
evaluation  and  revision  tasks  were  highly  structured  through  the  use  of 
specific  procedures  and  software  which  supported  and  enforced  these 
procedures. 

The  software  tools  developed  under  this  project  are  intended  to 
support  a  broad  spectrum  of  tutorial  and  drill  and  practice  tasks. 

They  would  not  be  as  useful,  for  example.  In  developing  CAI  to  simulate 
equipment  or  processes.  Similarly,  the  authoring  system  developed  is 
text  oriented  as  opposed  to  supporting  extensive  production  of  computer¬ 
generated  graphics.  Unless  sophisticated  qraphics  production  software 
and/or  equipment  is  available  (a  requirement  beyond  the  scope  of  this 
project),  the  production  and  delivery  of  computer  graphics  is  quite 
expensive.  Additionally,  when  use  of  supplementary  hard  copy  visuals 
(e.g.,  schematics  and  photographs)  is  considered,  there  are  relatively 
few  tutorial  or  dril 1-and-practice  applications  in  which  computer  gen¬ 
erated  graphics  can  be  shown  to  be  cost  effective. 

Project  Context:  The  Advanced  Instructional  System 

The  authoring  support  software  was  designed  for  application  in  the 
context  of  the  AIS  located  at  Lowry  Air  Force  Base,  Colorado.  The  AIS 
is  ideal  for  this  purpose  since  it  is  a  large  scale  computer-based  in¬ 
struction  (CBI)  system  supported  by  hardware  and  software  designed  to 
support  both  CMI  and  CAI.  The  system  was  designed  to  improve  the  effec¬ 
tiveness  and  efficiency  of  Air  Force  technical  training  and  to  provide  an 
operational  research  facility  for  assessing  innovations  in  instructional 
technology.  The  system  supports  four  technical  training  courses  repre¬ 
sentative  of  the  range  of  cognitive  and  performance  skills  required  by 
enlisted  Air  Force  personnel.  An  adaptive  instructional  decision  model 
employs  state-of-the-art  computer  hardware,  software,  statistical  method¬ 
ologies  and  instructional  procedures  to  provide  instructional  management 


and  individual  tied  assignments  to  alternat  t  vo  instructional  materials. 

A\x  yours?  Structure.  I  act)  A  l  S  tours?  Is  divided  into  "blocks"  of 
instruction  win  tit  nvt\  reguire  from  1  to  10  days  to  complete.  lach  It  look 
contains  a  miml'pr  of  lpssons  and  a  comprehensive,  end-of-Mock  test. 
Within  a  Nook,  lpssons  arp  arranood  in  a  hierarchy  Itasrd  on  thoir  pre- 
reguisite  relationsltins.  A  typical  Itiprarchy  resontltlos  a  spt  of  pa  rail  pi 
chains  diverging  and  converging  on  cprtain  pivotal  lpssons,  and  a  studpnt 
mat  alternately  work  on  lpssons  in  two  or  morn  parallol  chains. 

iho  haste  unit  of  instruction  is  the  lpsson.  I  act)  losson  ionsists 
of  a  sot  of  ot'.irt  t  i  vps  ,  two  or  morn  parallpl  forms  of  a  criterion 
referenced  tost,  a  criterion  defining  adeguate  mastery  on  thp  test,  and, 
t.vpicallt  a  self-tost  In  which  students  can  pvaluate  their  under¬ 
standing  of  the  lesson  before  taking  the  criterion  test. 

A  lesson’s  instruction  is  provided  by  one  or  more  modules,  each  of 
which  teaches  the  complete  lesson  content.  When  two  or  more  modules  are 
present,  they  represent  alternative  instructional  treatments,  depend¬ 
ing  on  the  lesson  content  and  the  nature  of  the  treatment,  a  module  may 
be  a  prograurted  text,  an  elaborated  technical  order,  an  audio-visual 
presentation,  or  given  thp  t'psults  of  this  prelect,  an  interactive  tAl 
sess i on . 

An  AlS  Student  Scenario.  A  student’s  first  experience  with  A i s  is 
to  complete  a  ('reassessment  battery  consisting  of  a  number  of  stales 
which  assess  cognitive  and  affective  factors  considered  to  be  predictive 
of  students’  performance  in  the  course,  lhe  student  then  roguests  an 
initial  assignment  b>  submitting  a  I  orward-icoi ng  Assignment  reguest  at 
a  management  terminal,  which  consists  of  an  optical  scanner  and  medium 
speed  printer.  At  this  point,  the  student  is  enrolled  in  the  course  but 
has  not  vet  entered  a  block  containing  actual  course  content,  lirst, 
therefore,  the  system  selects  t.hp  block  in  which  the  student  is  to  start 
work.  Since  the  student  lias  not  yet  completed  any  course  work,  only 
Mocks  which  have  no  preregul sites  are  considered.  If  there  is  more 
than  one  such  block,  the  one  containing  the  fewest  students  relative  to 
the  desired  number  in  that  block  is  selected,  lhe  student  is  then 
assigned  to  an  appropriate  learning  center  and  home  carrel  and  to  a 
specific  lesson,  module,  and  criterion  fpst. 

lessen)  assignment  decisions  at-e  made  jointly  by  two  major 
components  of  the  Svstem--the  Adapter  and  the  Resource  Allocation  Model . 
lhe  Adapter  attempts  to  select,  for  each  assignable  lesson,  the  one 
module  that  is  most  appropriate  for  that  student,  lhis  decision  can 
lie  based  on  a  variety  of  rules,  e.g.,  selec  t  the  module  which  the 
student  is  predicted  to  complete  in  the  shortest  tine  tasscrilng  the 
student  is  also  predicted  to  pass  the  criterion  test),  tach  alternative 
module  is  given  a  weight  indicating  its  relative  ('reference,  lhe 
Resource  Allocation  Model  assigns  preference  weights  to  modules  on  the 
basis  of  minimising  the  impact  of  the  assignment  on  the  availability  of 


instructional  resources.  The  final  lesson  and  module  selection  is  based 
on  a  compromise  between  the  two  sets  of  preference  weights.  The  form  of 
the  criterion  test  is  chosen  at  random. 

The  student,  after  receiving  the  first  assignment  printout  (called 
a  Student  Status  Report)  at  the  management  terminal,  reports  to  the 
learning  center  instructor,  obtains  the  instructional  resources  required 
for  the  assigned  module,  and  begins  work,  normally  at  a  home  carrel. 

After  studying  the  lesson  materials,  the  student  completes  a 
multiple-choice  seif- test  and  reviews  the  material  pertaining  to  any 
questions  answered  incorrectly.  The  student  then  completes  the  lesson 
criterion  test  and  submits  the  test  form  to  a  management  terminal.  The 
resulting  Student  Status  Report  details  the  student’s  performance  on 
the  criterion  test  (percentage  total  score,  items  missed,  objectives 
failed,  and  pass/fail  decision)  and  the  next  assignment.  If  the  test 
criterion  was  not  met,  the  student  is  reassigned  the  same  lesson  and  an 
alternate  form  of  the  test.  Otherwise,  the  lesson,  module,  and  test 
selection  procedures  are  repeated,  and  the  student  is  assigned  a  new 
lesson. 

If  the  student's  assignment  is  a  CAI  module,  there  is  only  a  slight 
variation  in  these  procedures.  The  function  of  the  self-test  is 
assumed  by  questions  embedded  in  the  CAI  presentation.  The  criterion 
test  is  also  administered  on-line  and  the  results  submitted  auto¬ 
matically  to  the  CM I  system.  The  Student  Status  Report  Is  displayed 
on  the  terminal  and  a  printed  copy  of  the  report  is  also  available  from 
the  management  terminal. 

When  the  student  has  completed  all  content  lessons  in  the  block, 
a  block  Review  lesson  is  assigned.  Following  review,  the  student  is 
randomly  assigned  one  of  the  alternate  forms  of  the  block  test.  While 
lesson  tests  can  be  viewed  as  diagnostic  tools,  end-of-block  tests 
serve  a  certification  function,  that  is,  since  there  is  no  end-of 
course  test,  block  test  performance  serves  as  the  basis  for  certifying 
mastery  of  the  objectives  contained  in  the  block.  A  student  not  meeting 
the  block  test  criterion  is  reassigned  to  the  block  in  a  status  whereby 
assignments  are  made  by  the  instructor  rather  than  by  the  system.  If 
the  block  decision  is  "Go,"  the  block  selection  loqlc  is  repeated,  and 
the  student  is  assigned  to  the  next  block  of  study.  The  student's  con¬ 
tinued  progress  through  the  course  is  essentially  a  repetition  of  these 
events. 
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design  of  a  cost-effective  C A 1  component  for  the  A1S  beoan  with  .in 
analysis  of  the  AIS  environment:  the  appropriate  role  of  iaI  within  a 
CM  I  systei.i  supportino  nilitarv  technical  training;  the  characterist  ns 
of  the  personnel  who,  it  was  anticipated,  would  he  developin']  t  A 1 
materials;  and  the  software  tools  currently  available  within  the  Als. 
Lessons  learned  from  prior  approaches  to  LAI  dove loimient  were  also 
considered. 

Tlwjtole  of  LA l  within  the  AIS 

first,  it  must  he  recoon i zed  that  the  prototype  Als  is  primarily  a 
compute r-nianaood  instructional  system,  Ihis  is  not  to  imply,  however, 
that  the  system  was  not  designed  to  acconnvdate  LAI.  lhe  S>stem 
lannuaoe,  LAMIL  (Computer  Assisted/Manaued  Instructional  Language) ,  was 
specifically  developed  to  support  hotti  LAI  and  LMI.  Rather,  within  the 
context  of  the  AIS,  LAI  was  seen  as  one  of  several  possible  media  avail¬ 
able  for  instructional  purposes.  Management  and  monitorin']  of  students’ 
prepress  throuoh  a  course,  assipnment  to  specific  instructional  treat¬ 
ments,  and  evaluation  of  instructional  effectiveness  are  all  supported 
by  the  LMI  component  of  the  system. 

(liven  the  nature  of  the  AIS  form  of  LMI,  definin']  the  role  of  lAl 
within  this  structure  was  relatively  straipht forward.  Recall  that  an 
AIS  course  is  divided  into  blocks  of  instruction  which  conclude  with 
certification  tests,  blocks  are  divided  into  lessons,  and  each  lesson 
is  supported  by  one  or  more  modules,  each  of  which  addresses  all  of  the 
lesson's  objectives.  Wiien  more  than  one  modulo  is  available  for  a 
lesson,  they  are  treated  as  alternative  inst ruct ional  treatments  for 
that  lesson.  The  Adaptive  Model  assures  that  a  student  is  not  assipned 
a  lesson  until  all  prerequisites  have  been  completed.  Assionmont  of  a 
specific  module  is  also  a  LMI  function.  Thus,  LAI  was  seen  as  providin') 
one  of  two  or  more  alternative  instructional  treatments  for  teaehinp 
a  lesson,  and  as  such,  i.Al  materials  would  be  packaped  aim  assipned  as 
modules.  Student  terminals  required  for  LAI  would  be  treated  as 
instructional  resources  manaoed  and  assigned  by  the  Adaptive  Model. 

It  was  assumed  that  if  a  student  was  assigned  and  completed  a  LAI 
module,  it  would  be  desirable  that  the  lesson  test  also  ho  administered 
on-line  rather  than  via  a  management  terminal  interaction.  While  the 
AIS  does  support  an  on-line,  computer-assisted  testum  (iAI)  capabiliti, 
it  was  designed  primarily  for  block  tests  and  was  not  totally  suited 
for  administration  of  lesson  tests.  Therefore,  a  lesson-level  testing 
capability  was  incorporated  into  the  LAI  component. 

The  next  major  question  concerned  the  tvprs  ot  applications  for 
which  these  relatively  expensive  modules  would  be  most  effective.  While 
the  use  of  LAI  for  normal  first -pass  instruction  was  one  obvious  answer. 
It  was  hypothesized  that  the  branching  capabilities  and  moment -to- moment 


control  over  student  behaviors  afforded  by  CAI  would  be  particularly 
useful  for  the  functions  of  review  and  remediation. 


As  AIS  courses  are  currently  structured,  each  block  ends  with  a 
review  lesson  which  is  assigned  inmediately  prior  to  the  block  test.  No 
specific  instructional  modules  had  been  developed  to  support  these 
lessons.  Rather,  it  was  intended  that  students  would  use  this  assign¬ 
ment  to  review  those  block  objectives  on  which  they  had  encountered 
problems.  The  CMI  system  provides  a  "Block  Report"  which  flags  the 
objectives  which  the  student  failed  on  the  first  attempt  at  the  lesson 
test.  Instructional  activities  during  this  period  are,  however, 
determined  primarily  by  the  student  and  the  instructor,  and  student 
performance  data  indicated  that  this  time  was  often  not  being  used 
effectively.  CAI  was  seen  as  an  excellent  way  of  remedying  this 
si tuation. 

A  procedure  was  envisioned  in  which,  when  a  student  was  being 
assigned  to  block  review,  the  Adaptive  Model  would  assess  the  student's 
prior  performance  in  the  block  and,  if  performance  was  found  to  be 
marginal,  assign  a  CAI  module.  The  CAI  module  would,  in  turn,  determine 
the  specific  objectives  on  which  the  student  had  encountered  problems, 
review  the  student  on  these  objectives,  administer  and  evaluate 
objective-level  diagnostic  tests,  provide  further  remediation  as 
necessary,  and  issue  a  Status  Report  suggesting  further  review  or 
assignment  to  the  block  test  as  appropriate. 

Block  remediation  presented  a  similar  situation.  A  student  who 
fails  a  block  test  is  placed  in  "block  remediation"  mode.  In  this  mode, 
the  System  accepts  any  lesson  tests  input  by  the  student  or  instructor 
input  of  a  second  attempt  block  test.  The  System  does  not.  However, 
make  any  specific  assignments.  This  role  is  delegated  to  the 
instructor.  To  guide  the  instructor  in  making  appropriate  assignments, 
the  objectives  which  the  student  failed  on  the  block  test  are  listed 
on  the  Student  Status  Report  printed  when  the  block  test  is  scored. 
Again,  student  performance  data  suggested  that  this  remedial  time  could 
be  employed  much  more  effectively. 

CAI  block  remediation  modules  were  envisioned  which  would  differ 
only  slightly  from  the  review  modules.  Student  assignments  to  CAI 
remediation  would  be  made  by  the  instructor,  and  selection  of  specific 
objectives  for  remediation  would  be  based  on  the  student's  block  test 
performance  rather  than  on  performance  in  the  block. 

Although  block  review  and  remediation  were  seen  as  two  prime 
targets  for  CAI,  it  was  assumed  that  CAI  would  also  be  used  for 
alternative  modules  for  first  pass  instruction.  While  CAI  might 
occasionally  be  used  for  the  first  module  developed  for  a  lesson,  as 
long  as  CAI  costs  remain  high,  it  was  expected  that  it  would  more  often 
be  used  as  an  alternative  treatment  designed  to  remedy  specific  problems 
detected  in  existing  modules. 


Anticipated  Character! sti cs  of  CM  development  Personnel 

In  considers  no  the  characteristics  of  the  personnel  who  would  be 
developin')  CAI  modules,  it  was  first  thouoht  that  a  team  approach,  such 
as  advocated  bv  Hunderson  (H7J),  would  be  appropriate.  Such  an  approach 
would  specify  differing  roles  for  (at  least)  subject  matter  experts,  in¬ 
structional  technolooi s ts ,  and  program  coders.  However,  further  analysis 
of  the  military  technical  training  environment  strongly  suggested  that, 
while  desirable,  the  team  approach  was  very  likelv  to  encounter  serious 
problems  in  practice. 

during  the  b  years  in  which  the  A1S  has  been  operational,  there 
have  been  repeated  efforts  to  define  specific  roles  for  the  specialists 
wno  are  so  important  to  effective  operation  of  CMJ  (e.g.,  materials 
writers,  evaluators,  and  data  base  managers).  To  gate,  these  efforts 
nave  had  only  limited  success.  It  was  concluded,  therefore,  that  the 
CAI  authoring  system  should  be  structured  so  as  to  allow  a  team 
approach  to  CAI  development  but  should  not  be  dependent  on  it. 

Tor  the  immediate  future,  at  least,  it  was  reasonable  to  expect 
that  the  personnel  wno  would  be  developing,  evaluating,  and  revising  CAI 
materials  would,  for  the  most  part,  bo  classroom  instructors.  It  was 
assumed  that  such  authors  would  typical lv  be  expert  in  their  subject 
matter  area,  would  have  limited  training  or  experience  in  instructional 
systems  design,  would  have  no  prior  exposure  to  CAI,  and  would  have  no 
computer  programming  experience.  Additional  pertinent  characteristics 
included  a  relatively  high  turnover  rate  for  military  instructors, 
little  or  no  opportunity  for  formal  training,  and  limited,  fragmented 
periods  of  availability  (e.g.,  oO  to  kKl  minutes  following  a  normal 
instructional  day). 

An  obvious  problem  raised  by  this  profile  ot  the  typical  CAI  author 
is  the  lack  of  computer  programming  expertise  and  the  strong  indication 
that  attempting  to  train  relatively  transient  authors  to  program  would 
not  be  practical.  Therefore,  an  approach  was  sought  which  would 
eliminate,  or  at  least  substantially  reduce,  the  need  for  computer 
programing  on  the  part  of  the  author. 

Another  implication  of  the  analysis  of  authoring-personnel 
characteristics  was  a  need  for  procedures  which  would  structure  the 
authoring  task.  At  the  same  time,  it  was  recognized  that  excessive 
structuring  could  be  perceived  as  being  undesirable,  even  offensive, 
and  could  consequently  detract  from  the  authors’  motivation.  It  was 
reasoned  that  they  would  be  more  amenable  to  task  structuring  once  CAI 
was  established  and  authoring  problems  had  been  recoqnized.  It  was  con¬ 
cluded  that  the  authoring  system  should  provide  a  degree  of  task 
structuring  necessary  to  prevent  a  novice  author  from  becoming  hope¬ 
lessly  bewildered  and  should  also  provide  for  implementing  more 
extensive  structuring  in  the  future. 
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The  high  turnover  rate  of  military  personnel  had  the  obvious 
Implication  that  the  authoring  system  and  procedures  should  be  such 
that  a  novice  author  could  quickly  be  brought  to  a  productive  level  of 
proficiency.  Further,  it  was  apparent  that  it  would  often  be  necessary 
for  new  authors  to  complete  CAI  modules  that  had  been  desiqned  and 
partially  developed  by  others. 

Coupled  with  the  hi qh  turnover  rate  was  the  expectation  that  there 
would  be  little  or  no  opportunity  for  formal  training  in  either  the 
mechanics  of  developing  CAI  materials  or  the  instructional  principles 
underlying  their  desiqn.  It  was  assumed  that  most  training  would  have 
to  be  conducted  on  the  job.  Thus,  self-instructional  methods,  with 
some  minimal  assistance  from  experienced  authors,  appeared  necessary. 

It  was  also  desirable  that  the  software  tools  themselves  be  self- 
documenting  and  self-instructional. 

Since  it  was  anticipated  that  authors  would  often  have  to  work  in 
short  segmented  periods  or  in  an  environment  subject  to  frequent  inter¬ 
ruptions,  it  was  desirable  that  they  have  rapid  access  to  the  portion 
of  the  module  on  which  they  were  working,  that  any  work  accomplished  be 
captured  immediately  rather  than  requiring  a  lengthy  storage  process, 
and  that  the  current  status  of  work  accomplished  be  clearly  summarized. 

Finally,  experience  gained  from  various  DOD  projects  indicates  that 
management  of  instructional  materials  development  is  an  area  that  has 
been  generally  problem  prone  because  of  its  complex,  unstructured  nature. 
Few  effective  management  procedures  have  been  successfully  implemented. 
Consequently,  little  monitoring  of  individual  authors  durinq  materials 
development  has  been  possible,  especially  with  respect  to  productivity 
and  quality  control,  thus,  there  was  a  need  for  software  tools  that 
would  facilitate  management  of  the  development  process.  As  was  tne 
case  for  structuring  the  authoring  task  itself,  it  was  necessary  that 
these  management  tools  be  fairly  open-ended  and  allow  for  further 
development  as  management  procedures  evolved.  The  review  of  completed 
materials  by  other  subject  matter  experts  was  known  to  be  particularly 
time  consuming.  Software  tools  which  would  structure  and  accelerate 
this  staqe  were  considered  especially  important. 

CAI  materials  evaluation  had  potential  for  problems  which  were  at 
least  equal  in  severity  to  those  of  authoring  per  se.  It  was  assumed 
that  the  CAI  authors  would  have  primary  responsibility  for  formative 
evaluation  of  their  own  materials  but  would  have  little,  if  any,  prior 
evaluation  experience.  As  a  result,  there  was  a  need  to  structure 
student  performance  data  collection,  retrieval,  and  reporting.  In  this 
case,  it  was  thouqht  that  there  would  be  little  negative  reaction  to 
over-structuring  and  that  the  data  collection  process  should  be  almost 
totally  predetermined  and  should  result  in  standard  reports  tailored  for 
formative  evaluation. 
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Sof  tware  i  ons  i  dor  a  t  ions 

The  two  aspects  of  the  A 1 S  software  wnich  most  strong ly  influenced 
authoring  system  design  decisions  were  aval  lain  1  i  t>  of  the  PA  UL 
programming  language  and  past  experience  with  the  use  of  interactive 
data  base  editors. 

The  _i  A‘!Il  Language.  CAMIL  (Computer  Assi sted /Managed  Instruct  ional 
Language!  (hTlasterer,  !'i/s)  is  a  higher  level,  general  Purpose  pro- 
granninj  language  developed  as  an  inteoral  part  of  the  AlS.  Several 
hundred  CAMIL  programs  are  currently  operational  and  are  used  for  all 
on-line  pnases  of  the  system. 

In  specifying  a  language  for  AlS,  it  was  determined  that  it  would 
need  to  he  both  C  A I  and  i_MI  oriented  and  usable  hv  personnel  with  a 
wide  range  of  experience.  It  support  reguirements  had  been  limited  to 
cAl  ,  a  language  such  as  TUTOR  (Sherwood,  1  "A 74)  as  implemented  at  the 
University  of  Illinois,  could  have  been  considered  while  traditional 
languages  such  as  PL/1,  FORTRAN  and  CObOl.  could  have  been  considered  if 
CMI  had  been  the  only  target  application.  Since  both  applications  were 
to  be  supported  and  no  known  language  contained  specific  capabilities 
for  both,  a  new  language  was  considered  necessary. 

To  accomplish  its  diverse  goals,  LAMU  is  divided  into  two  sub¬ 
languages:  basic  CAMIL  and  l x tended  iAMII.  basic  lAMli  is  oriented 
primarily  toward  systems  and  applications  prooranriino  tasks.  It  pro¬ 
vides  a  full  range  of  standard  data  types  including  1N1LC.LR,  dUMUIR, 
lOilUAl  ,  CHARACTER,  STRINu,  Tt \T  ,  LI  ASS  and  SIT.  The  data  structuring 
methods  include  ARRAY  and  RLLOKO  types.  Ihe  flavor  of  the  procedural 
component  is  similar  to  other  ALiiOL -based  languages:  procedures  are 
easily  defined;  the  scope  of  variables  is  controlled  through  the  use  of 
blocks;  and  statements  can  be  combined  into  logically  meaningful  groups 
through  the  use  of  compound  statements. 

I xtonded  CAM1L  is  oriented  toward  the  rapid  development  of  large 
programs.  Its  most  important  feature  is  the  sentence  facility  winch 
allows  coding  in  natural  language-1  ike  statements.  Also,  fewer  pro¬ 
grammer  coiments  are  required  because  of  the  level  of  documentation 
which  the  sentence  facility  provides.  i*v  using  sentence  declaration 
statements,  new  sentences  can  be  created  by  defining  new  words--nouns , 
verbs,  adverbs,  adjectives,  and  prepositions.  These  words  can  be 
combined  in  any  meaningful  way  to  form  a  large  set  of  sentences.  New 
data  types  and  operators  can  also  be  defined.  Sentences,  extended  data 
types,  and  extended  operators  can  then  be  placed  in  a  CAMIL  library. 
Lxtonded  CAMIL  Programs  can  also  utilize  basic.  iAMIL,  allowing  the  user 
as  much  control  as  experience  permits. 

A  total  system  approach  was  adopted  to  make  the  language  reliable, 
efficient,  and  easy  to  use.  This  system  consists  of  the  language,  a 
compiler,  loader,  interpreter,  source  program  editor,  data  base  file 


manager,  and  program  archiver.  The  combination  of  language,  compiler, 
and  source  program  editor  facilitates  efficient  creation  and  mainten¬ 
ance  of  reliable  CAMlt  programs  and  reliability  Is  enforced  during 
execution  bv  the  Interpreter.  Programs  Interface  easily  with  the  file 
manager  to  rapidly  perform  the  extensive  data  base  management  required 
for  CM1  applications. 

Hue  to  the  nature  of  the  CAM  11  system,  large  amounts  of  error-free 
code  can  be  written  and  checked  out  in  a  relatively  short  time,  uiven 
the  structured  nature  of  the  language,  the  enforced  modularity,  and  the 
interactive  environment,  relatively  few  logic  errors  are  encountered. 

Once  a  program  has  been  compiled,  it  is  almost  certain  to  run  properly. 
Finally,  the  t \ tended  CAM1L  sentence  facility  provides  a  powerful  pro¬ 
gramming  and  documentation  tool  for  relatively  Inexperienced 
programmers . 

A I S  Lxperience  with  Interactive  Kilters.  While  the  AIS  K-U  functions 
are  supported  by  xAMli  software,  the  day-to-day  operation  of  the  OH 
system  is  controlled  through  a  set  of  data  base  editors.  Ihe  intent  of 
these  editors  was  to  allow  course  personnel  with  no  programming 
language  skills  to  define  the  characteristics  of  their  courses  ano 
establish  the  rules  bv  which  student  assignments  are  made. 

Tho  approach  taken  was  th.it  of  a  totally  table-driven  system.  At 
each  point  where  an  instruct ional  decision  is  required,  an  attempt  was 
made  to  allow  for  a  variety  of  decision  rules.  Tne  indication  of  which 
rule  was  to  bo  used  (and  in  sow  cases,  the  rule  itself)  was  then 
treated  as  a  data  item  rather  than  being  coded  in  the  software,  lhus, 
both  the  decision  rule  and  the  information  to  be  processed  by  tie  rule 
(typically  a  combination  of  student  performance  data  and  course 
character! st  ic  parameters)  are  considered  to  be  data.  With  tms 
approach,  software  changes  are  only  required  when  the  basic  operating 
philosophy  of  the  svstem  is  altered.  Non*:1  operational  changes  in 
course  content  and  configuration,  resource  inventories,  and  student 
assignment  selection  rules  are  made  by  changing  the  course  data  base  via 
the  Interactive  data  base  editors. 

Operational  experience  with  this  approach  has  been  quite  positive. 
Relatively  few  software  changes  have  been  necessary  to  moot  the 
system's  evolving  instructional  requirements,  despite  the  complexity 
of  the  data  base,  course  personnel  have  been  able  to  use  the  editors 
to  institute  data  base  changes  appropriate  to  their  needs.  Ihis  is 
not  to  say,  however,  that  the  use  of  the  editors  is  totally  simple  and 
straightforward.  The  data  base  and  the  interactions  among  its  com¬ 
ponents  are  complex.  In  some  cases,  this  complexity  was  not  adequately 
taken  into  consideration  In  the  human  engineering  of  the  editors,  while 
all  of  the  editors  prompt  the  user's  input  to  sow  degree,  extensive 
prompting  was  sometimes  sacrificed  in  favor  of  efficiency.  In 
retrospect,  this  trade-off  was  often  inappropriate. 
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In  general,  however,  the  interactive  editor  approacn  was  thought  to 
hold  considerable  promise  for  facilitating  at  least  some  aspects  of  the 
JAI  development  process.  It  was  also  recognized  that  where  such  an 
approach  was  adopted,  it  would  be  desirable  to  provide  extensive 
prompt i  mi. 

Prior  Approaches  to  tne  Authoring  Problem 

A  growing  recognition  of  the  problems  associated  with  C A 1  develop¬ 
ment  has  resulted  in  a  substantial  literature  addressing  these  problems 
ano  proposing  alternative  solutions.  The  following  sections  provide  a 
brief  introduction  to  the  concept  of  authoring  systems,  an  overview  of 
tne  types  of  existing  systems,  and  a  summary  of  two  Air  Force 
experiences  in  CAI  development. 

Authoring  System  considerations  and  criteria,  tinn  (1974)  lists 
four  criteria  for  assessing  tne  ef fecti veness  and  utility  of  CAI  author¬ 
ing  languages:  reliability,  efficiency,  flexibility,  and  convenience. 
Inese  same  criteria  can  be  applied  to  the  more  general  concept  of 
authoring  systems. 

diulor  reliability,  Jinn  includes  automatic  recovery  for  both  author 
am;  student  following  system  failure,  limiting  tne  loss  of  authored 
material  and  limiting  the  domain  of  author  errors,  i.e.,  an  error  in 
one  part  of  a  program  should  not  impact  other  parts  of  the  program  or 
ether  programs,  efficiency  refers  to  both  the  time  required  for 
authoring  and  the  computer  time  required  to  translate  author  language 
statements  into  executable  code.  Flexibility  considerations  include 
access  to  a  variety  of  devices,  access  to  alternate  modes  of  execution 
or  conventions  and  the  capability  of  adding  new  operators,  statements, 
atu;  subroutines. 

Author  convenience  is  treated  as  a  major  consideration.  Jinn 
suggests  that  the  language  (or  system)  should  have  a  minimum  of 
redundancy  and  irrelevant  syntax,  e.g.,  the  program  listing  should  be  no 
more  complex  than  the  author's  actual  task.  There  should  be  provision 
for  alternative  authoring  styles,  e.g.,  while  many  authors  indicate  a 
preference  for  interactive  entry,  others  prefer  to  work  with  paper  forms 
which  remind  the  author  of  system  capabilities  and  requirements.  Witn 
respect  to  revising  an  existing  program.  Jinn  notes  the  advantages  of 
on-line  editing  and  suggests  that  the  system  should  provide  access  to 
the  original  file,  use  straightforward  notation  for  determining  changes 
to  be  made,  and  confirm  that  changes  were  accomplished.  In  testing  a 
program,  the  author  should  be  able  to  begin  execution  at  any  point  and 
trace  through  the  program  using  labels  as  indicators  of  location.  The 
language  notation  should  help  a  reviewer  understand  the  intent  of  the 
instructional  content  and  strategy.  Finally,  access  to  system 
capabilities  should  increase  with  experience. 

kaplow  (197b)  states  that  in  order  to  maximize  assistance  to  the 


author,  it  is  not  sufficient  to  simply  add  authoring  aids  to  a  pro¬ 
graming  language.  Rather,  the  total  CA1  system  must  be  organized 
around  this  goal.  He  then  describes  what  he  considers  to  be  the  basic 
features  of  such  a  system. 

First,  the  system  should  provide  a  structured  format  to  help 
authors  organize  their  concepts.  The  structural  units  should  match  the 
author's  conceptual  units  and  impose  a  degree  of  modularity.  The 
current  working  unit  should  always  be  identified  and  the  author's  state¬ 
ments  should  refer  only  to  this  unit.  Kaplow  suggests  that  the 
operational  aspects  of  the  authoring  and  CAI  delivery  components  should 
be  separated.  A  CAI  program  should  be  treated  as  a  data  base  containing 
the  content  and  structure  of  the  program  as  well  as  the  computer 
commands  to  be  executed.  The  system  design  snould  make  it  explicit  that 
the  program  is  a  collection  of  information  organized  so  as  to  be 
amenable  to  understanding.  The  computer  itself  should  automatically 
perform  many  programming  functions,  such  as  checking  structural  complete 
ness  and  finding  cross-reference  errors.  Finally,  the  system  should  not 
reguire  that  a  program  be  complete  before  it  can  be  tried  out. 

In  his  subsequent  discussion,  Kaplow  makes  a  number  of  additional 
critical  points.  The  system  should  be  tolerant  of  user  errors  and  pin¬ 
point  errors  at  the  time  they  are  made.  The  detailed  actions  to  be 
taken  at  student  run  time  should  be  defined  on  the  basis  of  the 
implications  of  the  author's  instructions  rather  than  having  to  be 
spelled  out.  Given  a  modular  proqram  structure,  the  system  should  help 
the  author  keep  track  of  the  interrelationships  between  the  various 
parts.  The  fact  that  it  is  often  easier  to  write  a  new  program  rather 
than  to  modify  an  existing  one  is  particularly  unfortunate  considering 
the  opportunity,  even  requirement,  for  CAI  revisions  based  on  student 
performance.  Since  the  source  of  this  problem  is  usually  one  pro- 
granner's  difficulty  in  understanding  the  structure  and  logic  used  by 
another,  the  system  should  place  particular  emphasis  on  the  ease  with 
which  one  can  build  on  existing  material. 

Example  Autnoring  Systems.  The  earliest  CAI  programs  were  written 
in  the  available  general  purpose  languages.  Although  this  is  still  a 
popular  approacn  (e.g.,  the  extensive  use  of  SASIC),  there  was  early 
recognition  that  authoring  could  be  facilitated  by  languages  tailored 
to  the  particular  requirements  of  CAI.  Consequently,  there  was  a  pro¬ 
liferation  of  CAI  authoring  languages--at  least  30  by  1973.  The  last 
decade  has  also  seen  experimentation  with  various  author  entry  systems-- 
approaches  which  are  relatively  independent  of  a  specific  language.  Tne 
following  paragraphs  briefly  describe  an  example  of  a  modern  CAI  author- 
inn  language  (TUTOR)  and  some  approaches  to  developino  autnoring  systems 

TUTOR  (Sherwood,  1974)  needs  to  be  considered  in  the  context  of  the 
complete  PLATO  (Programing  Louie  for  Training  Operations)  system.  The 
system  provides  for  interactive  entry,  easy  trial  and  revision  of  code. 


and  is  quite  responsive  to  authors'  needs  in  terms  of  display  time, 
compilation  time,  and  diagnostics.  The  author's  task  is  facilitated 
t>y  a  number  of  aids,  such  as  on-line  access  to  reference  materials, 
sample  program  routines,  and  files  of  current  documentation  and  comments. 
To  a  large  extent,  the  PLATO  approach  to  authoring  is  based  on  the  model 
of  an  autnor  who  is  a  versatile  professor,  an  export  in  the  subject 
■wtter,  an  experienced  teacher  with  sound  but  innovative  ideas  about  in¬ 
structional  presentation,  and  a  capable  programmer .  Within  the  uni¬ 
versity  environment,  this  approach  has  resulted  in  a  large  number  of 
excellent  LAI  lessons.  In  other  environments,  where  the  authors  have 
been  less  experienced,  less  skilled,  and/or  less  motivated,  the  approach 
has  not  proven  as  satisfactory . 

Jowsey  (1974)  describes  five  categories  of  approaches  to  building 
easy  author-entry  systems:  separation  of  logic  and  content,  avoidance 
of  anv  authoring  language,  use  of  lesson  planning  guides,  conversational 
materials  generation,  and  macro  systems.  The  approaches  in  each 
category  tend  to  ouild  on  the  concepts  of  t ne  prior  categories. 

Almost  all  of  these  approaches  employ  the  tactic  of  separating  in¬ 
structional  content  from  program  logic.  Tnis  divides  the  authoring  task 
in  a  way  that  is  particularly  amenable  to  a  team  approach.  Dowsey  notes 
tnat  to  oe  effective,  such  separation  requires  similarity  of  structure 
between  the  two  components. 

The  no-author-languaqe  approach  not  only  separates  content  and  logic 
but  does  not  require  the  author  to  define  the  logic.  Only  the  content 
is  specified,  in  the  form  of  frames  or  problem  categories.  Tnis  is  then 
acted  upon  by  a  lesson  generation  program  which  treats  the  content  as 
data  to  produce  instructional  materials. 

The  use  of  lesson  planning  guides,  while  requiring  the  services  of 
a  coder,  permits  the  author  to  communicate  with  the  computer  in  Inglish. 
Typically,  the  author  defines  the  material  to  be  presented,  questions, 
expected  answers,  and  corresponding  courses  of  action  on  a  standard  form 
which  specifies  the  categories  of  information  required.  One  of  the  more 
sophisticated  examples  of  this  approach  is  Dowsey’s  own  i.OUR$tMAk.tK , 
designed  for  use  with  the  COURSLWRITER  III  language.  It  is  based  on  use 
of  a  paper  form  which  includes  a  presentation  section  containing  the 
material  to  be  displayed,  a  guestion  section  indicating  the  student 
response(s)  expected,  a  decision  section  defining  whether  a  branch  is  to 
be  taken  (and  if  so,  the  type  of  branch),  and  an  analysis  section  con¬ 
taining  response  judging  rules. 

Conversational  materials  generation  represents  a  quite  different 
approach.  The  interactive  system  assists  the  author  by  eliciting  the 
content  and  logical  structure  of  the  lesson  through  a  natural  language 
conversation.  Paloian  (1974)  describes  one  example  of  this  approach. 

The  author  defines  the  program  structure  by  entering  a  sequence  of  action 
verbs.  The  system  checks  the  accuracy  of  the  sequence  and,  if  it  is 
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correct,  prompts  the  author  to  supply  the  text,  anticipated  responses, 
counter  names,  etc. 

Mowsey  sees  the  use  of  macro  routines  as  potentially  ueino  the  most 
powerful  approach.  In  usino  such  a  system,  the  author  retrieves  a  pre- 
progranined  sequence  of  code  and  specifies  arouments  which  complete  the 
routine.  One  characteristic  of  this  approach  is  that  it  imposes  a 
definite  structure  on  the  material  produced.  The  Time-shared,  Inter¬ 
active,  Computer-Controlled,  Information  Television  (T1CC1T)  system 
(Stetton,  Volk  .«  Buruierson,  Id/.!)  is  probably  the  best  known  example  of 
a  macro  system.  T1CC1T  employs  a  sinple  instructional  stratooy,  oriented 
toward  concept  learnino,  which  assumes  learner  control  over  sequence. 

1  ho  strategy  specifies  how  tne  subject  matter  should  be  structured  u'.d-» 
rules,  examples,  practice  items)  and  even  suggests  the  appropriate 
number  of  items  in  each  cateoorv.  Thus,  much  of  the  author's  task 
consists  of  molding  the  content  to  fit  tne  strategy.  Planning  guides 
are  used  to  format  the  content  and  define  presentation  tactics  by 
selecting  among  available  options.  A  macro  processor  turn  converts  this 
information  (content,  content  format,  and  strategy  options)  into  computer 
languaoe. 


One  other  macro-oriented  approach  wnicn  deserves  mention  is  the 
use  of  Monoforms  ^schulc,  19/p)  developed  for  use  by  military  authors 
on  PL-JO  IV.  The  Monoform  macros,  written  in  IbTdil,  were  intended  to 
facilitate  preparation  of  frequently  used  question  types  by  providing 
a  question  format  and  eliminating  the  need  for  the  author  to  understand 
TUTOR.  A  total  of  nine  macros  were  developed  for  multiple  choice, 
constructed  response,  and  matching  ouestions.  ..ifiiri  each  question  type, 
the  Mono forms  differ  with  resnoct  to  variations  in  format,  type  of 
feedback  and  (for  multiple  choice  questions),  order  of  alternative 
presentation.  There  are  vilso  a  number  of  options  within  each  Monoform. 
The  author  copies  the  desired  Monoform  and  follows  tne  instructions 
(supplied  in  the  form  of  urogram  comen ts)  to  supply  the  question  content 
and  tailor  the  TIMOR  connands  to  n is,  her  specific  regui remen t s .  Schul.t 
reports  tnat  use  of  Monoforms  reduced  the  /  to  u  hour  question  develop¬ 
ment  time  to  onlv  10  to  Is  minutes. 


Air  force  OAI  Authoring  J  xnerience.  (iimwich  (1°.’  ’)  reports  a 
comparison  cTF  Tltili  and  I'l^VT 0  authorTno  efficiency  conducted  .it  Maxwell 
Alt!.  The  results  of  the  comparison  were  inconclusive  with  little 
difference  found  between  the  two  approaches.  In  the  production  of  V 
student  contact,  hours  of  bAl,  tne  PLATO  team  required  an  average  of 
manhours  per  dour  while  tne  TkYIT  team  required  .Me.  Ot  greater 
interest  is  tiinwicn’s  description  of  tne  procedures  toll  owed  and 
problems  encountered,  particularly  by  the  I’lAlM  team.  Trio  training  pro¬ 
vided  for  t'LnTO  team  members  consisted  of  an  intensive  .  week  session 
and  subsequent  continuous  consulting  support  py  tne  I'LAlP  staff.  Hull 
team  training  began  with  i  weeks  of  tami 1 iari cat  ion  follov.eo,  some  months 
later,  by  .!  weeks  of  intensive  training.  In  both  cases,  tne  training 
required  appears  excessive  to  tne  authors  ot  Inis  report.  <  wither. 
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nimwich  reports  that  in  many  instances  the  authors  did  not  capitalize 
on  PLATO'S  t'lexihi  1  i  tv  with  resnect  to  instructional  stratenies  and, 
consequently ,  uni  not  demonstrate  the  system's  full  capabilities. 

Hallman,  JeLeo,  Main  and  ..illnan  (107?)  provide  a  comprehensive 
description  of  an  experiment  in  the  use  of  PLATu  IV  for  Air  Force 
technical  trainino.  At  tiie  beoinninq  of  the  test,  conducted  at  Cnanute 
AFp,  the  typical  PLATO  autnnrino  model  was  adopted  with  each  author 
action  independently.  Learninq  Ti.THR  was  found  to  occupy  a  major  portion 
of  authors'  tine.  Authors  nau  varyinq  styles  and  quality  control 
stanuarus,  and  is  a  result,  the  curriculum  was  fraunenteu  witn  little 
continuity  between  lessons,  dal  1  man  et  al.  concluded  that  a  basic  flaw 
in  tins  approach  was  tne  unrealistic  assumption  that  all  materials 
authors  were  exports  in  both  subject  matter  and  instructional  practices. 

A  team  approach  was  subsequently  adopted  m  winch  tne  team  con¬ 
sisted  of  an  author,  subject  matter  expert,  instructional  proqranrier , 
and  coder.  Althouqh  this  was  a  distinct  improvement  over  the  prior 
approach,  a  number  of  problems  were  still  encountered.  <o  written 
procedures  were  defined,  only  informal  understanuinis  anonq  team  members. 
Tnis  resulted  in  time  cons uni  no  coordination  Problems  and  inefficient 
use  of  team  specialists.  Administrative  and  manaoement  procedures  were 
never  well  defined  and  there  was  a  continuous  need,  never  resolved,  for 
better,  more  extensive  author  traininq. 

A  number  of  PLAT.)  features  were  found  to  be  quite  useful.  On-line 
data  collection  routines  supplied  by  the  PLATO  staff  and  the  capability 
for  on-line  text  editing  were  both  considered  important.  TUTOR  was 
adequate  for  the  site's  needs  with  authors  handliivi  the  simpler  aspects 
of  proqramninq  and  coders  required  for  only  the  more  complex  portions. 
Only  a  few  of  the  more  experienced  authors,  however,  capitalized  on 
PLATO's  instructional  flexibility.  Almost  all  of  the  lessons  produced 
employed  the  same  simple  tutorial  model.  The  report  authors  suqoest 
tnat  this  approach  was  followed  because  tne  materials  were  easy  to 
prepare,  the  subject  matter  was  not  that  complex,  and  student  com¬ 
prehension  and  retention  requirements  were  low.  branching  was  used 
mainly  for  forced  review  and  TUTOR'S  response  judo  inn  capability  was 
seldom  utilized.  Only  about  00  percent  of  the  questions  developed 
employed  a  constructed  response  format.  Not  only  were  constructed  re¬ 
sponse  questions  more  time  consuminq  to  code,  students  disliked  them 
because  of  unfamiliarity  with  the  typewriter  keyboard. 

Hallman  et  al.  drew  a  number  of  conclusions  relevant  to  the  current 
project.  The  team  process  was  found  to  be  more  efficient  and  effective 
tnan  individual  authors.  Authors  did  not  exploit  PLATO's  full 
capabilities  due  to  resource  constraints,  lack  of  CA1  expertise,  and 
inadequate  traininq  in  instructional  proqrarriinq  techniques.  Finally, 
sophisticated  DM  capabilities  nay  not  be  necessary  for  effective  learn- 
inq  of  the  type  of  tasks  and  level  of  knowledqe  required  for  adequate 
military  technical  traininq. 


The  types  of  problems  encountered  at  Maxwell  and  Chanute  appear 
typical  for  military  technical  training.  Kimberlin  (1977),  describing 
the  status  of  Project  ABACUS  at  Ft.  Gordon,  reports  b  to  0  month  slips 
in  the  full  implementation  of  CAI  courses.  The  major  problems  were  re¬ 
ported  to  be  changes  in  the  Plans  of  Instruction  during  CAI  development 
and  the  fact  that  the  project  was  never  assigned  an  adequate  number  of 
instructional  programmers. 

Design  Conclusions 

Although  many  aspects  of  the  military  technical  training  environ¬ 
ment  are  not  amenable  to  efficient  development  of  instructional 
materials,  particularly  CAI,  it  was  concluded  that  it  was  preferable, 
at  this  time,  to  adapt  the  system  to  the  environment  rather  than  to 
develop  an  authoring  system  which  assumed  a  more  responsive  environment. 
It  is  hoped  that  the  authoring  tools  would  themselves  act  as  chanqe 
agents.  For  example,  it  was  decided  that  the  authoring  system  should  be 
designed  so  as  to  promote  the  concept  of  an  authoring  team  but  not  to 
rely  on  its  existence.  Given  a  trade-off  between  authoring  flexibility 
and  a  structured  presentation  format,  structure  was  almost  always 
selected.  Uiff !cul ties  in  assuring  adequate  author  training  was  accepted 
as  a  major  problem.  Despite  the  fact  that  many  features  of  CAMIL  were 
desiqned  expressly  for  this  purpose,  it  was  recognized  that  any  approach 
which  relied  cn  an  author/prograimier  had  little  chance  of  success. 
Further,  it  was  considered  unlikely  that  personnel  would  be  assigned  to 
such  a  specialized  function  as  CAI  computer  programming  in  sufficient 
numbers  to  adequately  support  many  authors. 

lhis  assessment  suggested  either  a  no-autnor-language  approach  or 
a  macro  system  approach.  The  former  was  considered  to  be  too  rigid  for 
the  evolving  computer-based  technical  training  environment.  A  macro 
system  modeled  on  the  TICCIT  approach  was  rejected  because  a  simpler 
authoring  process  was  desired;  the  sane  objection  was  encountered  for 
approaches  similar  to  Mono  forms.  The  adopted  design  combined  the  best 
aspects  of  several  prior  approaches,  relied  neuvily  on  characteristics 
of  the  existing  CAMIL  system,  and  capitalized  on  prior  AIS  experience 
wi  tii  interactive  data  lias.1  editors.  As  suggested  by  kaplow  ( 1  '♦/’;>) ,  the 
total  system  was  designed  with  the  major  goal  of  facilitating  CAI 
materials  development. 

To  alleviate  the  need  for  computer  programming,  the  approach 
adopted  was  to  provide  a  small  number  of  flexible  "template"  CAI  pre¬ 
sentation  programs,  each  of  which  would  support  i  class  of  CAI  nodules. 
While  this  approach  still  requires  the  services  of  a  computer  programmer 
to  expand,  the  extent  of  these  services  was  reduced  in  two  wavs.  First, 
the  template  programs  were  designed  to  bo  as  flexible  as  possible  with¬ 
out  being  unduly  complex.  Principles  which  lud  been  employed  in 
developing  tin*  /US  Adaptive  Model  and  its  data  base  were  used  to  assure 
that  a  single  program  structure  would  support  a  variety  of  nodules 
addressing  different  topics.  Second,  given  the  characteristics  of  tne 


CAM  I L  language  and  a  tempi  at  e  program  which  was  well  structured  and 
documented ,  It  was  anticipated  that  a  relatively  inexperienced  programmer 
could  modify  the  basic  program  to  meet  the  regui cements  of  new 
appl  ications. 

The  major  emphasis  was  placed  on  designing  an  interact  ve  authoring 
editor  by  means  of  which  an  author  could  define  C  A I  content  and  branching 
logic.  The  design  goals  were  to  provide  authors  with  a  variety  of 
options  within  a  structured  framework,  to  appropriately  ouit.e  and  prompt 
authors’  decisions  with  respect  to  these  options  so  as  to  minimise  the 
need  for  formal  training,  to  automatical  1 v  accomplish  as  much  of  the 
prograiaiing  detail  as  feasible,  and  to  eliminate  any  need  for  authors  to 
be  aware  of  the  computer  language.  The  format  ot  the  authors’  input 
was  to  be  as  similar  as  possible  to  what  students  would  actually  see. 
Author  errors  were  to  he  detected  and  identified  at  the  lime  they  were 
coiini t ted.  Standard  routines  and  reports  were  to  be  provided  tor  student 
performance  data  collection  and  analysis.  I  mall.,  the  system  was  to 
provide  convenient  access  to  meaningful  information  whivh  would  assist 
m  managing  lAl  development. 
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1 1 1  APPROACH 


The  approach  taken  to  supporting  efficient  CAI  production  in  the 
technical  training  environment  centered  around  development  of  a  CAI 
Authoring  Editor  and  a  template  Presentation  Program.  Student  perfor¬ 
mance  data  acquisition  routines  were  built,  into  the  presentation  program 
and  reports  developed  which  focus  on  formative  evaluation  requirements. 

A  CAI  Materials  Print  Program  was  developed  to  provide  hard  copy  list¬ 
ings  of  CAI  materials  for  author  and  student  use.  All  but  one  of  the 
supporting  programs  were  written  in  CAM1L.  The  Editor  and  Presentation 
Program  were  designed  to  operate  on  the  current  A1S  interactive 
terminals,  which  are  a  modification  of  the  PLATO  terminal,  but  provisions 
were  made  for  easy  transition  to  less  expensive  terminals.  The  Authoring 
Editor,  Presentation  Program,  Data  Acquisition  and  Analysis  Programs, 
the  Materials  Print  Program,  and  documentation  of  the  authoring  system 
are  described  in  detail  in  the  following  sections. 

] ne  CAI  Authoring  Editor 

The  heart  of  the  software  supporting  tno  AIS  CAI  authoring  system 
is  an  interactive  Authoring  Editor,  lo  introduce  the  Editor,  the 
structure  and  characteristics  of  the  CAI  modules  produced  with  it  are 
first  described.  The  Editor  itself  is  then  discussed,  beginning  with 
various  summary  displays  which  provide  central  reference  points  tor  the 
author's  work.  The  mechanics  of  generating  text  and  question  frames 
and  of  defining  branching  logic  are  then  described  and  illustrated  in 
some  detail. 

Module  Structure,  frame  Types  and  frame  Hags.  As  structured  by 
the  Authoring  Editor,  a  CAI  module  is'  segmented  into  objectives,  through 
the  Authoring  Editor,  the  author  can  define  only  the  sequence  in  which 
objectives  are  to  be  presented.  Any  branching  between  objectives  is  a 
function  of  the  Presentation  Program.  Each  module  must  include  an 
Objective  0  that  contains  material  (usually  an  overview)  pertaining  to 
the  lesson  as  a  whole.  The  author  may  then  define  up  to  100  numbered 
objectives  containing  the  module's  instructional  content.  Each  objective 
can  contain  up  to  100  frames.  Ten  frame  types  have  been  defined  which 
can  be  classified  into  three  categories:  textual  content  frames; 
question  frames;  and  special  purpose  frames,  lew  types  can  be  defined 
as  requirements  arise.  The  frame  types  in  each  category  are  desci ibed 
briefly  below. 

There  are  six  different  types  of  textual  content  frames.  Those  are 
all  similar  in  that  they  simply  present  textual  information  to  the 
student  and  require  no  response  other  than  an  indication  that  the  student 
is  ready  to  proceed.  The  six  types  are  differentiated  primarily  for 
management  reasons.  In  some  cases,  a  specific  frame  type  may  be  re¬ 
quired  at  a  certain  point  in  the  frame  sequence  and  listing  frames  by 
type  provides  a  quick  overview  of  tno  instructional  sequence. 


Tne  basic  "Text"  frame  is,  as  its  name  implies,  the  primary  medium 
for  presenting  instructional  content,  a  Text  frame  can  contain  up  to 
four  "pages"  or  screen  displays.  There  is  no  provision  for  branching 
within  frames,  i.e.,  between  pages,  unce  a  frame  has  been  selected,  all 
of  the  pages  in  that  frame  are  presented.  Authors  are  instructed  that 
a  Text  frame  should  present  a  discrete  chunk  of  information  pertaining 
to  a  single  concept. 

Tne  second  textual  frame  type  is  the  Elaboration  frame.  It  differs 
from  a  Text  frame  only  by  name  and  intended  function--to  present  infor¬ 
mation  to  students  whose  performance  indicates  that  they  need  further, 
more  detailed  explanation  of  a  concept  presented  in  a  Text  frame.  The 
Title  frame,  when  used  for  the  purpose  implied  by  its  name,  can  be 
recognized  as  a  dividino  point  in  the  frame  sequence.  The  Statement  of 
Objective  frame  is  regui red  at  the  start  of  each  objective  other  than 
Objective  i).  S  i  m  i  1  a  Fly ,  an  Overview  frame  is  mandatory  within  Objective 
J.  Finally,  a  Materials  List  .rame,  listing  any  reference  materials 
which  the  student  needs  to  complete  the  module,  is  required  in  Objective 
0.  Any  of  the  mandatory  frame  types  can  also  be  used,  at  the  author's 
discretion,  at  other  points  in  the  module. 

The  Editor  currently  supports  two  types  of  Question  frames: 
multiple  choice  and  constructed  response.  These  frame  types  provide  the 
primary  means  of  evaluating  students'  understanding  of  the  material  being 
presented.  Question  frames  consist  of  the  question  stem,  up  to  five 
mill tiple-cnoice  alternatives  or  nine  anticipated  constructed  response 
alternatives,  feedback  statements  for  each  alternative  and  prompt  state¬ 
ments  associated  with  successive  attempts  to  answer  the  question. 

There  are  two  special  purpose  frame  types,  neither  of  wnich  is  pre¬ 
sented  to  students.  The  first  is  documentation  frame  required  at  the 
beginning  of  each  objective  to  provide  a  means  of  documenting  the 
authoring/evaluation/revision  process.  The  author  is  instructed  to  pro¬ 
vide  information  such  as  the  dates  during  which  the  module  was  being 
developed  or  revised,  the  names  of  additional  authors  who  have  worked  on 
the  module,  the  learninq  strategies  employed,  the  nature  of  and  reason 
for  any  revisions,  etc.  It  is  anticipated  that  the  information  requested 
will  become  more  structured  as  a  function  of  experience.  The  second 
special  purpose  frame  type  is  the  "Branching  Decision"  frame.  It  is  a 
dummy  frame,  containing  no  material,  which  allows  for  a  branching  point 
without  anythinq  being  presented  on  the  screen. 

There  are  eiqht  varieties  of  "flags"  which  can  be  set  aoainst  a 
frame  to  indicate  to  the  CAI  Presentation  Program  that  a  particular 
action  is  to  be  taken  when  that  frame  is  encountered.  Like  the  frame 
types,  flags  have  one  to  two  character  code  names  which,  when  displayed 
on  the  frame  listing,  contribute  to  providing  a  sui.tnary  of  the  instruc¬ 
tional  strategy  employed. 

Four  of  the  flags  (Objective  Passed,  objective  Failed,  Lesson 
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Passed,  and  Lesson  Failed)  notify  the  Presentation  Program  that,  if  this 
frame  is  encountered  due  to  author-defined  branchino,  the  student  has  or 
has  not  completed  the  objective  or  lesson  satisfactorily.  The  Skip  This 
Frame  flag  indicates  that  the  frame,  even  when  encountered,  is  not  to  be 
presented  to  students.  This  allows  the  author  to  store  alternative 
material  in  the  body  of  the  module  itself.  The  Fix  Answer  List  flag, 
which  can  only  be  set  against  a  multiple-choice  question  frame,  indi¬ 
cates  that  the  list  of  alternatives  is  to  be  presented  in  the  order  input 
by  the  author  rather  than  being  randomized.  A  Student  Break  flag  elicits 
a  display  generated  by  the  presentation  program  suggesting  that  the 
student  take  a  short  break  from  working  on  the  module.  It  allows  the 
author  to  alleviate  student  fatigue  on  lonq,  complex  modules.  Finally, 
the  Decision  Point  flag  indicates  that  the  presentation  program  is  to 
collect  Decision  Point  student  performance  data  following  presentation 
of  that  frame. 

Module,  Objective  and  Frame  Summary  Displays.  Throughout  the 
Lditor,  a  philosophy  was  adopted  of  providing  extensive  prompting  and 
"fail-safe"  mechanisms.  Lach  author  entry  is  prompted  to  the  extent 
possible.  Commands  which  will  result  in  the  deletion  of  materials  or 
branching  Ionic  require  a  second  "Yes,  I  want  to  do  it,"  response  on 
the  part  of  the  author  and  it  is  always  possible  to  back  out  of  an  error 
situation  without  the  Editor  taking  any  action. 

Whether  an  author  wisnes  to  create  a  new  CAI  module,  revise  an 
existing  module  or  display  a  module,  the  same  basic  procedures  apply. 
After  accession  the  Authoring  Editor  from  an  interactive  terminal,  the 
author  encounters  a  display,  illustrated  in  Figure  1,  requesting  the 
sequence  of  numbers  (Course  dumber  through  Module  Dumber)  identifying  tne 
particular  module  to  be  accessed  or  created.  Entry  of  each  value  is 
prompted  and  tne  range  of  allowable  values  is  specified.  The  author  mav 
also  request  a  list  of  existing  modules  (by  pressing  Function  Key  FI), 

If  this  ootion  is  selected,  the  author  is  asxed  whether  all  mouules  or 
only  those  belonging  to  a  particular  author  are  to  be  listed  and  wno trier 
the  list  is  to  be  displayed  on  the  terminal  or  printed  on  the  central 
line  printer. 

If  the  identifier  of  an  existing  module  is  entered,  tne  author  is 
asked  whether  that  module  is  to  be  displayed,  charmed  (revised  or  adued 
to),  or  deleted.  Whereas  any  module  can  be  uisplayed,  only  those  created 
by  tnat  author  can  be  changed  or  deleted.  If  the  author  enters  the 
identifier  of  a  module  which  does  not  exist,  tne  author  is  asked  wnetner 
a  new  module  is  to  be  created,  and  if  so,  whether  the  module  is  to  he 
a  copy  of  an  existing  module.  This  copy  feature  allows  an  author  to 
revise  a  module  which  nas  already  been  implemented  in  tne  classroom,  to 
revise  another  author's  module,  etc.  If  the  new  module  is  to  be  created 
from  scratch,  the  author  is  required  to  enter  the  title  oe  tne  module 
and  tne  number  of  objectives  which  it  is  to  contain. 


If  the  author  is  creating  a  new  nodule  or  revising  an  existing 


CAI  Authoring  Editor 


Module  Description  Information 
(Press  Fj  for  CAI  Module  Listl 


1  Course  number  0 

C  Course  version  0 

3  Block  number  0 

4  Lesson  number  0 

5  Nodu 1 e  number  0 


Course  number 
Range  is  0  thru  192  3 


Figure  1.  Authoring  Lditor  Module  Selection  Display. 


module,  the  next  display  asks  which  data  collection  routines  are  to  be 
activated:  response  data,  decision  point  data,  and/or  student  comments. 
None,  any,  or  all  of  the  three  types  can  be  selected. 


The  author  next  encounters  the  Objective  List  display  illustrated 
in  Figure  2.  This  display  provides  a  central  reference  point  for  the 
author--a  top  level  overview  of  the  module's  content  and  the  control 
point  for  creating,  changing,  or  listing  complete  objectives.  Objectives 
against  which  no  material  has  been  entered  are  shown  as  being  undefined 
(Undef).  For  the  display  illustrated  in  Fiqure  2,  the  nodule  includes 
the  mandatory  Objective  0  and  four  additional  objectives,  two  of  which 
are  as  yet  undefined. 

Actions  which  can  be  taken  at  this  point  are  listed  as  Options  at 
the  bottom  of  the  display.  If  the  author  enters  the  number  of  an  objec¬ 
tive,  the  list  of  the  frames  defined  for  that  objective  is  displayed. 

One  can  also  add  or  delete  an  objective,  reorder  the  sequence  in  which 
the  objectives  are  to  be  presented,  or  request  a  printer  listing  of  one 
or  more  objectives.  As  is  the  case  for  all  of  the  Authorino  tditor  dis¬ 
plays,  the  author  can  back  out  without  taking  any  action  by  pressing  the 
"BACK"  key. 

If  a  new  module  is  being  created,  all  of  the  objectives  would  be 
shown  as  undefined,  and  the  author  would  first  be  required  to  define 
Documentation,  Overview,  and  Materials  List  frames  for  Objective  0.  The 
author  could  then  define  additional  Objective  0  frames  or  return  to  the 
Objective  List  display  and  proceed  to  define  content  and  branching  with¬ 
in  the  individual,  numbered  objectives. 

When  the  author  accesses  a  specific  objective,  the  first  display 
encountered  is  the  Frame  List  illustrated  in  Figure  3.  The  Frame  list 
provides  a  central  point  for  work  within  an  objective.  Tin'  list  defines 
the  sequence  and  types  of  frames  comprising  the  objective  and  provides 
an  overview  of  the  branching  logic  and  any  special  conditions  imposed  on 
specific  frames.  Actions  which  can  be  taken  are  listed  as  Options  at 
the  bottom  of  the  display. 

A  Frame  List  consists  of  one  to  four  pages  witn  no  to  30  frames 
listed  per  page.  The  objective  represented  by  the  Frame  List  snown  in 
Figure  3  is  relatively  short,  only  22,  frames.  Tne  display  provides  a 
substantial  amount  of  information  about  the  content  and  instructional 
strategy  employed  in  this  objective.  Lach  frame  has  both  a  number  0 
through  Jo)  am;  a  name  consisting  of  one  or  two  letters  (o.g.,  0  for 
documentation,  S  for  Statement  of  Objective)  and  one  or  more  digits 
indicating  whether  this  was  the  first,  second,  etc.  frame  of  inis  tyne 
to  be  defined.  If  there  is  branching  logic  associated  with  a  frai.u  , 
the  class  of  logic  (Pre-frame,  After-frame,  or  Response  contingent)  is 
indicated,  and  limits  on  the  number  of  attempts  allowed  to  answer 
questions  are  shown,  as  are  frame  Hags. 
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OPTIONS:  Objective  number,  display /chance  frame  list 
I,  insert  objective 
R,  reorder  objective 
0,  delete  objective 
BACK,  return  to  module  selection  pace 
ENTER  choioe  > 


Figure  l.  Authoring  Editor  Objective  List  Display. 
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C ,  comment  file 

♦/-,  display  next /prev 1 ous  frame  list  page 
BACK,  return  to  objective  page 
ENTER  choice  > 


Figure  J. 


Authoring  Editor  Frame  List  Display  Showing 


Set  of  Initial  Options. 


J1 
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Tni’  oojec  1 1  vo  i  1  lustra  toil  in  l  iq.iro  '  be  Mils  with  the  require*, 
documentation  frame  (.'1},  a  State! ion t  o*  objective  xsl  ' ,  a  Title  iTil) 
jiui  an  overview  frame  (  1).  iollowinq  a  sir, ole  Text  fra  >e  tTl),  tne 
author  tests  the  student  on  a  prerequisite  concept,  usinq  two  multi. mo 
choice  questions  i  V*1  ano  qM.  ) .  onl\  two  attempts  arc  allowed  to  answer 
eacn  question.  The  second  question  is  *ol lowed  e>  ter-fra  ie)  brancn- 
l no  looic  which  routes  the  student  around  a  review  of  fie  conce  >t  t frames 
■  through  1.’)  if  performance  on  the  two  mestions  is  adequate.  dnl  •  too 
existence  of  the  loqic  is  snown,  not  its  oetail.  The  review  consists  of 
three  Text  frames  aiu;  two  more  nultiole  choice  questions.  .<ote  that  the 
nano  of  frane  number  10  is  Tld.  This  inJicates  that  tne  frame  was  added 
later,  a*ter  frane  Til  had  been  defined.  The  frame  1-'  wfter-'rane  looic 
determines  wiietlier  the  student  nas  now  mastered  tne  concert.  1*  not,  tne 
student  is  returned  to  frane  number  ..  and  to  review  tin:  concert  a  second 
tine,  when  t rame  number  11  is  encountered  tne  second  tine,  t  ie  Ye-tra'ic 
looic  tn  passes  the  nultirle  choice  questions  aru;  routes  tne  stuuent  to 
fra"ie  n under  Id.  lo  determine  now  many  students  are  ueino  routed  t  irouun 
these  review  frames  and  the  tine  required  for  review,  t'ie  author  nas  de¬ 
fined  decision  Point  Plans  (,')  prior  to  if  raw  number  ana  toll  own  no 
( fran«>  nuntier  Id)  the  review  sequence. 

The  '-win  bony  of  instructional  content  is  presented  im  frane  nu  > 
uors  Id  tnrouon  do,  consisting  of  Text,  l laborat ion  it),  and  constructed 
response  pi)'  francs.  Since  no  specific  limit  has  been  placed  on  toe 
number  of  attempts  for  answerinq  tneso  questions,  the  default  value  of 
three  attempts  will  be  allowed,  Response  umtinqent  branenino  Ionic  tas 
o*  eii  defined  for  frame  ]i'l.  If  tno  resoonse  matches  a  particular  amt - 
ci pateo  incorrect  response,  tne  student  will  be  reuteu  to  tlaboration 
frame  tl  uesioned  to  correct  tne  misconception,  omen. iso,  frame  tl  is 
sri ppeo  via  too  ,H  1  »f ter- frame  branch. 

lasterv  of  tin.  objective  is  evaluated  e>  six  lultiple  choice 
questions  l frame  numbers  cl  throuon  A*).  Only  one  attempt  is  allowed  on 
each  question.  Tne  criterion  for  passinq  tne  objective  is  tour  out  of 
six  correct,  and  tne  author  nas  defined  After-frame  branchiuq  looic 
followin')  ;)?!.»  to  Sxio  tne  last  two  questions  if  tlie  first  tour  wore  all 
correct.  Otherwise,  the  student's  performance  is  evaluated,  following 
frame  .,’010.  If  the  criterion  was  not  met,  frame  Tk>  is  presented  to  in¬ 
form  the  student  of  this  fact.  The  frame  Hap  (Oi  )  indicates  tne 
occurrence  of  an  objective  failure  to  the  presentation  pro«iram.  If  the 
criterion  was  net,  after  either  four  or  six  questions,  frame  T10  is 
presented  wliicn  conqratul ates  the  student  and  tne  01'  Mao  notifies  the 
Presentation  Prouran  that  the  objective  has  been  passed. 

The  actions  which  the  author  can  take  are  listed  as  Options  at  tne 
bottom  of  the  display  (see  Fiqure  .5).  tntrv  of  a  soeeific  frame  mntvr 
accesses  that  frame  for  the  author’s  review  or  revision.  Option  I  (In¬ 
sert)  allows  the  author  to  define  a  new  frame  and.  insert  it  at  any  point 
in  the  frame  sequence.  If  the  Insert  Option  is  selected,  tlie  author  is 
asked  to  provide  tne  number  of  the  frame  after  which  the  new  frame  is  to 


be  inserted  and  to  select  the  Tyne  of  the  new  frame.  The  Trane  Type 
selection  nenu  is  shown  in  Figure  4.  Once  Frame  Type  nas  been  selected, 
the  author  is  placed  in  the  frame  creation  node  for  that  type,  dote 
that  Option  11  on  the  Frame  Types  menu  allows  the  author  to  copy  an 
existing  frame.  Any  frame,  including  one  from  another  author's  program 
can  be  copied.  Further,  the  author  nas  the  choice  of  making  an  actual 
copy  of  the  frame,  wnicii  can  then  be  modified  to  suit  tne  author's 
purposes,  or  of  simply  "aliasing"  an  existin']  frame.  .With  the  latter 
option,  the  frame  is  not  actually  copied  but  rather,  wnen  an  "alias" 
frame  is  encountered  by  the  Presentation  Prooram,  the  frame  content  is 
retrieved  from  its  original  location  for  display. 

The  third  initial  Frame  List  Option,  Specify  Flaos  (F),  allows  tne 
author  to  set  the  previously  discussed  Frame  Flags  aoainst  specific 
frames.  Specify  Branching  Logic  (B),  permits  the  autnor  to  define 
branchino  logic  for  suecific  frames.  Option  C,  Comen t  Tile,  accesses 
the  list  of  student  comments  which  have  been  made  against  specific 
frames.  Tne  existence  of  such  comments  is  indicated  by  the  occurrence 
of  a  "C"  in  the  Flans  column  of  the  Frame  List  display.  The  Frame  List 
may  consist  of  up  to  four  panes,  anti  the  control  characters  and 
access  display  of  the  next  or  previous  pane  of  the  List,  respectively. 

The  delete  Frame  (d)  Option  allows  deletion  of  one  or  more  frames  which 
are  no  longer  required.  The  Reorder  Frames  (J)  Option  permits  tne  author 
to  move  frames  to  different  positions  in  the  *rane  sequence.  If  the 
author  decides  to  replace  all  of  a  frame's  content,  the  Replace  Frame 
(R)  Option  is  available.  Finally,  the  author  can  always  return  to  the 
Objective  List  Jisolay  by  press  inn  the  BACK  key. 

Text  Generation.  Tne  process  by  w.oich  all  six  types  of  textual 
content  frames  are  produced  and  edited  is  essentially  the  same.  Wnen 
the  author  elects  to  create  one  of  these  frame  types  via  the  Frame  Types 
menu  (see  Figure  4),  a  blank  template,  such  as  is  illustrated  in  Figure 
u,  is  displayed.  Wote  tnat  the  frame  number,  frame  name,  and  number  of 
turns  page  witnin  tne  frame  are  shown  at  the  top  of  the  display  as  is 
the  fact  that  the  author  is  in  Insert  rode  and  has  already  entered  one 
line.  Typed  content  appears  on  the  line  next  to  the  arrow-soaped  Cursor, 
and  tne  WLXT  key  is  pressec  to  end  each  line.  Tne  autnor  is  free  to 
enter  the  content  in  any  format  within  tne  constraints  of  the  margins, 
indicated  by  tne  cursor  on  tne  left  and  the  vertical  line  on  the  right, 
eacn  page  can  contain  up  to  LI  lines  of  ob  characters  oacn. 

After  entering  one  or  more  characters,  the  autnor  can  press  eACh 
to  access  a  collection  of  edition  functions.  This  situation  is  ill us- 
trated  in  Figure  o  and  is  the  same  as  the  dis.lav  which  woulu  be  pre¬ 
sented  if  the  autnor  accessed  an  existing  Text  *rare  to  modify  it.  Tne 
Editin'1  Options  are  listed  at  the  bottom  of  the  display. 

The  Insert  Lioe(s)  (I)  Option  allows  the  author  to  define  additional 
lines  of  text  following  or  between  lines  of  existin'!  text.  While  insert- 
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FRAME  TYPES: 


1  Documentat ion  7  Constructed  Response  ^Kjestion 

2  Statement  of  Objective  8  Branching  Decision 

3  Overview  9  Title 

4  Text  If  Material  List 

5  Elaboration  11  Cops'  of  an  existing  frame 

6  Miltiple  Choice  Question 

ENTER  choice  > 


Figure  4.  Authoring  Lditor  Frame  List  Display 
Showing  Frame  Type  Selection  Options. 
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l  These  are  the  first  coupT#  of  lines 

>  typed  by  the  author. 


OPTIONS:  Enter  text  material. 

Press  NEXT  to  end  each  line. 
COPY  keys  copy  previous  line. 
Press  BACK  when  f i n i shed . 


1  Lines 


Figure  j.  Authoring  Editor  Text  Frame  Display  as  Seen  When 

Creating  a  Frame. 
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1  These  ere  the  first  couple  of  lines 

2  typed  by  the  author. 


OPTIONS:  I,  insert  line(s)  R,  replace  line 

D,  delete  line(s)  B,  draw/erase  box (esi 
S,  save  line(s)  F,  fetch  saved  line(s) 

E,  empty  save  buffer 

♦/-,  display/change  next/previous  page  of  frame 
BACK,  return  to  frame  selection 
NEXT,  display/change  text  for  next  frame 
ENTER  Choice  > 


Figure  6.  Authoring  Editor  Text  Frame  Display  Showinq 
Frame  Editing  Options. 
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iny  material  between  lines,  the  two  lines  preceding  and  one  line 
followiny  the  entry  are  also  displayed. 

The  Replace  Line  (R)  Option  allows  the  author  to  edit  one  or  more 
existiny  lines.  The  oriyinal  version  of  the  line  is  displayed,  the 
author's  new  version  is  shown  below  it  (to  the  right  of  the  cursor)  and 
the  precediny  and  followiny  lines  are  also  displayed  to  provide  the  con¬ 
text  of  the  chanye.  A  situation  in  which  line  3  of  a  display  is  beiny 
edited  is  illustrated  in  Tiyure  1.  A  set  of  Ldit  keys  facilitate  the 
euitiny  process  by  allowiny  the  author  to  erase,  copy,  or  temporarily 
store  the  complete  line,  specific  words  or  individual  characters.  After 
the  line  nas  been  changed,  the  iditor  redisplays  the  edited  text  and 
Moves  down  to  the  next  line  to  allow  the  author  to  edit  it.  As  in  the 
Insert  mode,  the  author  can  PACK  out  of  Replace  mode  at  any  time. 

The  delete  Line(s)  (d)  option  (see  1  inure  o)  allows  the  author  to 
erase  one  or  more  lines  of  text.  The  Iditor  prompts  the  author  to  in¬ 
dicate  the  lines  to  be  deleted  (the  numbers  of  the  first  and  last  lines;, 
surrounds  the  indicated  lines  by  a  box  for  visual  emphasis  and  asxs  if 
the  author  wishes  to  complete  or  cancel  the  deletion  request.  Any  text 
on  the  pane  followin'!  the  deletion  is  moved  up  to  replace  the  deleted 
material . 

The  Save  Line(s)  (,S)  Option  provides  a  capability  for  saving  up  to 
..’1  lines  of  material  in  a  "Save"  buffer.  The  saved  material  can  then  be 
inserted  at  any  point  in  any  textual  content  frame  in  any  LAI  nodule. 

This  is  useful  in  editing  material,  corvine  pant's  of  frames  without 
retypin'!  and  moving  material  to  other  modules.  Unco  an  autnor  lias  saved 
one  or  more  lines,  they  can  be  retrieved  at  anv  time  until  the  Save 
buffer  is  emptied. 

lne  letch  Saved  Line(s)  (I  1  Option  provides  the  means  hv  which 
material  is  retrieved  from  the  Save  buffer.  If,  when  the  material  is 
fetched  to  be  inserted  on  a  pane  already  containin'!  text ,  the  inserted 
material  will  result,  in  a  total  of  more  than  J1  lines,  the  author  is 
warned  of  this  fact  and  given  the  option  of  cancel  inn  the  letch  command 
or  allowin'1  tne  Iditor  to  reformat  the  text  to  overflow  onto  subsequent 
pages  of  the  frame. 

me  nyl,  have  .offer  (1)  Option  simply  erases  tne  contents  of  the 
Save  buffer  •rior  to  saving  new  material. 

ini'  or. iw -O  rase  'ox(es)  (l’«)  Option  allows  the  author  to  urapn- 
lcul  1  1  su'vnund  one  or  more  lines  of  material  for  erpnasis.  /in  example 
is  snown  m  li  tire  .  mixes  can  be  ui'.iwn  around  am  sin.le  line  or 
!ivup  Ur  lines.  1  he  autnor  enter',  the  number  u!  tne  first  ami  last 
lints  to  Pi  surrounded  and  ..lie  horizontal  width  ot  tne  Lo\  i s  determined 
tv  the  position  of  tne  first  and  last  characters  in  idle  lines.  Op  to 
sixteen  boxes  can  ee  defined  in  a  smim  (rare. 
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Th  is  is  an  example  of 

This  is  an  example  of 


REPLACE  MOPE 
a  mi  ape  lied  word 

a  missp 


2  Ln 


that  need  editing. 


OPTIONS:  Enter  *ext  material  revision. 

Press  NEXT  to  end  each  line. 

COPY  keys  copy  line  to  be  modified. 
Press  BACK  when  finished. 
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Bomb  fuses  are  classified  in  ThREE  WAYS  -  -  by 
POSITION,  by  FUNCTION,  and  by  METHOO  OF  ARMING 


-*-*  By  position  relates  to  the  POSITION  the 
f use  will  occupy  in  the  bomb . 


A  fuse  can  be  of  three  position  t 


o  a  NOSE  fua^e  if  it  is  used 
in  the  FRONT  of  the  bomb. 

o  a  TAIL  fuse  if  it  is  used 
in  the  REAR  o'"  the  bomb. 

o  MULT  I POSITIONAL  if  it  can 
be  used  in  EITHER  nose  or 
tail  POSITION. 


OPTIONS:  I,  insert  line (si  R,  replace  line 

0,  delete  line (si  B,  draw  erase  box  lesl 
S,  save  line  (si  F,  fetch  saved  li  nets’! 

E,  empty  save  buffer 

♦  display  change  next  previous  page  of  frame 
BACK,  return  to  frame  selection 
NEXT,  display  change  text  for  next  frame 
ENTER  Choice  > 


fi oure  o.  Authoriiio  [ilitov  Text  I  v.vno  i'lsplov 
Showing  I'so  of  lloxos  for  implmsis. 


The  final  three  options  pertain  to  the  mechanics  of  pane  and  frame 
selection.  To  move  forward  to  the  next  pane  of  the  current  frame  or  back 
to  the  previous  pane,  the  author  enters  or  respectively.  Press¬ 
ing  the  NLXT  key  routes  the  author  to  the  first  pane  of  the  next  frame  in 
the  sequence.  The  HACK  key  returns  the  author  to  the  pane  of  the  frame 
List  display  listing  the  current  frame, 

Althouqh  the  Authorino  Lditor  does  not  currently  provide  any  exten¬ 
sive  graphics  capability,  the  auttior  can,  with  some  ingenuity,  create 
simple  graphics  figures  by  using  the  standard  keyboard  characters  and 
any  "unrecognized"  character,  which  creates  a  solid,  character-sized 
rectangle.  An  example  is  provided  in  Figure  y.  Note  that  the  display 
is  shown  in  Inspect  Only  node  with  an  abbreviated  list  of  author  options. 

Question  Generation.  Multiple  choice  and  constructed  response 
question  frames  provide  the  author's  primary  means  of  interacting  with 
and  evaluating  the  student's  understanding  of  the  material  being  pre¬ 
sented.  As  structured  by  the  Authoring  lditor,  a  multiple  choice 
question  consists  of  the  question  stem  and  two  to  five  alternatives. 

After  selecting  this  frame  type  (QM)  from  the  frame  Types  menu,  the 
author  is  shown  a  template  display  and  prompted  to  enter  the  question 
stem.  The  text  of  the  stem  is  displayed  as  'it  is  entered  and  a  double 
press  of  the  HINT  key  notifies  the  lditor  that,  the  stem  lias  been  com¬ 
pleted.  The  author  is  then  prompted  to  enter  the  first  al ternative. 

A  double  NLXT  indicates  the  end  of  each  alternative  and  elicits  a  prompt 
for  the  next  alternative.  This  process  continues  until  tin'  author  in¬ 
dicates  that  all  alternatives  have  been  entered  by  pressing  BACK  or  until 
the  maximum  of  five  alternatives  lias  been  entered.  An  example  of  a 
partially  completed  multiple  choice  question  is  shown  m  figure  Id. 

Once  the  question  stem  arnl  alternatives  nave  been  entered,  the 
author  is  prompted  to  specify  the  correct  alternative  or  alternatives. 

At  least  one  correct  answer  must  be  defined  before  the  author  is  allowed 
to  exit  the  question  frame.  In  addition,  the  author  is  prompted  to 
specify  the  maximum  number  of  attempts  allowed  in  answering  the  question. 
The  author  may  select  a  number  nr  indicate  that  the  default  value, 
currently  three,  be  used. 

A  constructed  response  question  consists  of  the  question  stem  and 
one  to  nine  anticipated  responses.  When  this  frame  type  (Pi)  is 
selected,  the  procedure  followed  is  essentially  the  same  as  for  a 
multiple  choice  question.  The  author  is  prompted  to  enter  tne  question 
stem  and  successive  anticipated  responses  up  to  a  maximum  of  nine.  The 
correct  resnonso(s)  and  maximum  number  of  allowable  attempts  must  then 
be  specified.  A  completed  constructed  response  question  is  illustrated 
in  figure  11.  The  asterisks  to  the  left  of  four  of  the  alternatives  in¬ 
dicate  that  any  of  these  responses  will  be  accepted  as  being  correct. 

Currently,  the  AIS  CA1  software  supports  only  exact  matches  on 
constructed  response  questions.  That  is,  otner  tnan  variations  between 
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OPTIONS:  ♦/-,  display  next,  previous  page  of  frame 

S,  save  line(s)  E,  empty  save  buffer 
BACK,  return  to  frame  selection 
NEXT,  display  text  for  next  frame 
ENTER  Choice  ^ 


Figure  9.  Authoring  Editor  Text  Frame  Display  in  Inspect 
Only  Mode  Showing  Use  of  Simple  Graphics. 
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three. 

OPTIONS:  Enter  multiple  choice  question  a  1 1 ernat i ves . 

Press  NEXT  to  end  each  line. 

R  line  with  no  material  (just  a  NEXT  kevi  will  allow 
question  alternative  entries. 

COPV  keys  copy  previous  line. 

Press  BflCK  when  finished. 


Fiqure  10.  Authorinq  Editor  Display  of  Partially 
Completed  Multiple  Choice  Question. 
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Figure  11.  Authoring  Editor  Pi  splay  of  Completed 
Constructed  Response  Question. 
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upper  and  lower  case,  the  student's  response  must  exactly  match  one  of 
the  author's  anticipated  responses.  Otherwise,  the  student's  answer 
Is  treated  as  an  unanticipated  response.  Work  Is  currently  in  progress 
to  support  recognition  of  key  words  and  strings  and  misspelled  words 
which  phonetically  match  an  anticipated  response. 

To  provide  students  with  substantive  information  about  their  re¬ 
sponses,  provision  has  been  made  for  author  entry  of  feedback  and  prompt 
statements  for  both  multiple  choice  and  constructed  response  questions 
(Options  F  and  P  In  Figure  11).  The  intent  of  a  feedback  message  is  to 
tell  the  student  that  his  or  her  answer  was  correct  or,  if  it  was  in¬ 
correct,  why  it  was  wrong.  For  each  alternative  or  anticipated  response 
defined,  the  author  can  write  a  feedback  statement  which  will  be  pre¬ 
sented  whenever  a  student  selects  that  alternative.  For  constructed  re¬ 
sponse  questions,  a  feedback  message  can  also  be  defined  for  the 
occurrence  of  unanticipated  responses.  If  the  author  does  not  define 
a  feedback  message  for  an  alternative,  a  standard  statement,  matched  to 
whether  the  response  is  correct  or  incorrect,  will  be  selected  at  random 
and  presented.  The  author  can  suppress  a  feedback  message  by  entering 
one  or  more  blank  characters. 

The  intent  of  a  prompt  message  is  to  guide  the  student  toward  the 
correct  answer.  Whereas  feedback  messages  are  alternative-specific, 
prompts  are  selected  and  presented  as  a  function  of  the  number  of  the 
student's  attempt  at  answering  the  question.  That  is,  prompt  message 
number  one  will  be  presented  following  the  student's  first  incorrect 
attempt,  prompt  two  following  the  second  incorrect  attempt,  etc.  Prompts 
are  not  presented  following  a  correct  response  but  are  otherwise  inde¬ 
pendent  of  the  specific  alternative  chosen. 

Both  feedback  and  prompt  messages  may  consist  of  up  to  three  lines 
of  60  characters  each.  Depending  on  the  author's  actions,  one  or  both 
messages  will  be  displayed  at  the  bottom  of  the  screen  following  a 
student's  response.  The  prompt  statement  immediately  follows  the  feed¬ 
back  statement,  giving  the  appearance  of  a  single  informative  message. 
This  concatenation  of  feedback  and  prompt  provides  the  author  with  a 
powerful  tool  for  responding  appropriately  to  students'  errors. 

Question  generation  mode  provides  the  author  with  some  of  the  text 
editing  features  (insert,  replace  and  delete)  described  under  Text 
Generation.  The  use  of  these  features  is,  however,  somewhat  restricted 
due  to  the  structured  nature  of  question  entry.  The  save  buffer  and  box 
options  are  not  available. 

Definition  of  Branching  Logic.  In  preparing  a  CAI  module,  defini¬ 
tion  of  effective  branching  logic  may  require  the  greatest  thought  and 
be  the  most  difficult  part  of  the  process  for  an  author  to  understand. 

The  effectiveness  of  a  lesson,  however,  can  hinge  on  now  well  this 
feature  is  used.  Therefore,  particular  attention  was  given  to 
facilitating  this  aspect  of  the  authoring  process. 


Within  the  AIS  CAI  scheme,  all  branchino  decisions  are  associated 
with  individual  frames  and  can  be  evaluated  at  three  different  points 
in  the  presentation  of  the  frame.  Pre-frame  Ionic  is  evaluated  at  the 
point  at  which  the  Presentation  Proqram  first  encounters  the  frame, 
before  it  is  displayed.  Response  logic,  which  can  only  be  defined  for 
a  question  frame,  is  evaluated  as  the  student  answers  the  question  with 
the  branch  being  taken  if  a  particular  response  is  made.  After-frame 
logic  is  evaluated  after  the  student  has  indicated  a  readiness  to  advance 
to  the  next  frame.  A  branch  can  be  made  to  any  frame  within  the  objec¬ 
tive  which  has  been  defined  in  the  Frame  List  display. 

Currently,  four  types  of  conditions  can  be  evaluated  by  Pre-  or 
After-frame  branching  logic:  (a)  Given  a  set  of  question  frames,  take 
the  branch  if  at  least  a  specified  number  of  these  questions  were  an¬ 
swered  correctly,  (b)  Given  a  set  of  question  frames,  branch  if  at 
least  a  specified  number  of  these  questions  were  answered  incorrectly. 

(c)  Branch  if  at  least  a  specified  number  of  a  given  set  of  frames  (of 
any  type)  have  been  presented,  (d'  Branch  if  at  least  a  specified  number 
of  a  given  set  of  frames  have  not  ueen  presented.  In  addition,  uncondi¬ 
tional  branches  can  be  defined  where  the  branch  will  always  be  taken  if 
the  branching  point  is  reached.  As  mentioned  above.  Response  logic  pre¬ 
sents  a  different  situation  in  which  the  branch  is  taken  if  a  particular 
response  is  made. 

Any  number  of  branching  logic  instructions,  of  any  or  all  of  the 
three  types,  can  be  entered  against  a  single  frame.  The  only  restriction 
is  the  total  storage  space  required  for  all  branching  instructions  within 
an  objective.  At  least  300  instructions  can  be  defined  for  a  single  ob¬ 
jective. 

Branching  instructions  are  evaluated  in  the  order  in  which  they  are 
encountered.  Thus,  whether  or  not  a  subsequent  instruction  will  be 
evaluated  is  dependent  on  the  result  of  evaluating  the  current  instruc¬ 
tion.  If  none  of  the  conditions  of  Pre-frame  logic  are  met  (or  if  eo 
Pre-frame  logic  was  defined),  the  frame  will  be  presented.  If  none  of 
the  conditions  of  Response  or  After-frame  logic  are  met  (or  if  no  such 
logic  was  defined),  the  next  frame  in  the  sequence  will  be  presented. 

The  process  by  which  an  author  defines  branching  logic  for  a  frame 
is  illustrated  by  Figures  12  through  2U.  Like  most  aspects  of  an 
author's  work  within  an  objective,  the  process  begins  on  the  Frame  List 
display  where  the  author  selects  the  Branching  Logic  Option  (B)  and 
enters  the  number  of  the  frame  for  which  logic  is  to  be  defined.  This 
results  in  the  Branching  Logic  display  shown  in  Figure  12.  In  the 
example  shown,  no  logic  has  yet  been  defined  for  the  frame.  The  author 
selects  the  action  to  be  taken  from  the  Options  1 i st — in  this  case,  to 
Insert  Frame  Logic  (I).  The  author  is_  then  asked  whether  Pre- frame. 
Response,  or  After-frame  logic  is  to  be  defined  and  a  brief  description 
of  each  of  the  three  types  is  provided.  Assume  that  After-frame  logic 
is  to  be  defined,  for  which  the  Author  enters  an  "A."  This  results  in 
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Branch! ng  Logic  for  Objective  1.  Frame  24 
_ Multiple  Choice  Qu eat  ton  8 _ 

Pre-Frame  Logic  none 


Response  Logic  -  none 


Pfter-Frame  Logic  -  none 


OPTIONS:  Frame  number,  disp' ay /change  frame  logic 
fCXT,  di splay /change  next  frame  logic 
I,  insert  frame  logic  D,  delete  frame  logic 
BPCK,  return  to  frame  list  page 
ENTER  choice  >  i 


Figure  12.  Authorinn  editor  Display  for  Defining 
Branchinq  loqic. 
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Fr— le  List  for  Entry  of  Logic  Condition 
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Enter  the  frame  number  to  be  presented  if  the  lo*ic 
conditions  specified  are  met.  NEXT  will  present  the 
current  frame. 

ENTER  choice  >  20 


Figure  U.  Authoring  tditor  Display  for  Selecting  To-be-brancheo 
to  Frame  for  Pre-  or  After-frame  Branching  Logic. 
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Frame  List  for  Entry  of  Logic  Condition 


1 

01 

21 

QM5 

2 

Ti  l 

22 

Qrn 

3 

Si 

23 

4 

01 

El 

qua 

5 

Ti 

25 

QM9 

6 

QM1 

26 

gni0 

7 

Qf12 

2/ 

T  9 

8 

T  2 

-e 

28 

Tl0 

9 

T  3 

126 

END  OBJ 

10 

T 1  2 

127 

END  LSN 

1  1 

gn  3 

12 

On  4 

1  3 

T  4 

14 

T5 

15 

QCl 

16 

El 

17 

T  6 

18 

T7 

19 

QC2 

20 

T  8 

Th« 

‘  frame  log’c 

to  be  evaluated  is  to  be  based  on  one 

of 
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following  conditions: 

C. 

i  f  a 

set 

of 

frames  (questions)  is  correct 

I. 

i  f  a 

set 

of 

frames  (questions)  is  incorrect 

P. 

i  f  a 

set 

of 

frames  has  been  presented 

N. 

i  f  a 

set 

of 

frames  has  not  been  presented 

NEXT,  no  conditions;  an  uncond it  tonal  branch  is  to  occur 
ENTER  choice  >  c 


Figure  14.  Authoring  Editor  Display  for  Selecting  Class 
of  Pre-  or  After-frame  Branching  Conditions. 
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Frame  List  for  Entry  of  Logic  Condition  Correct 
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Frames  which  oust  meet  the  conditions  specified  will  be 
denoted  by  an  *  to  the  left  of  the  frame  number.  To 
change  a  designation,  reenter  the  frame  number.  To 
denote  additional  frames,  enter  the  frames  numbers  and 
an  •  will  appear.  Press  NEXT  when  finished. 

ENTER  frame  number  >23 


Figure  15.  Authoring  Editor  Display  for  Selecting  Question 
Frames  to  be  Evaluated  for  Correctness  fo**  a 
Pre-  or  After-frame  branch. 
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Frag x  List  for  Entry  of  Logic  Condition  Correct 
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Enter  the  number  of  frames  which  must  meet  the 
condition  specified  in  order  to  present  the  desired 
frame. 


ENTER  choice  >  4 


Fiqure  16.  Authoring  Editor  Display  for  Indicating 
Number  of  Correctly  Answered  Questions 
to  Meet  Pre-  or  After-frame  Branching  Condition 
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Branching  Logic  for  Objective  1,  Frame  24 
_ Multiple  Choice  Question  8 _ 

Pre-Frame  Logic  -  none 


Response  Logic  -  none 


After-Frame  Logic 

1  Go  to  Tit  i f  al 1  of  the  following  are  correct: 
0M5.QM6.0M 7 .QMS 


OPTIONS:  Frame  number,  disp' ay/change  frame  logic 
NEXT,  display/change  next  frame  logic 
It  insert  frame  logic  D,  delete  frame  logic 
BACK,  return  to  frame  list  page 
ENTER  choice  > 


Figure  17.  Authoring  Editor  Display  Showing  Completed 
After-frame  Branching  Instruction. 
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Frame  List  for  Entry  of  Response  Logic 
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Enter  the  frame  to  be  presented  correspond 1 ng  to  the 
alternative  chosen.  I f  no  response  logic  is  to  be  used 
for  the  alternative,  then  press  NEXT. 

ENTER  frame  number  for  alternative  2  or  NEXT  >  26 


Figure  lb.  Authoring  Editor  ui^oiay  for  Defining  Response 
Contingent  Branching  Logic. 
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Branching  Logic  for  Objective  2,  Frame  24 
_ Multiple  Choice  Question  6 _ 

Pre-Frame  Logic  -  none 

Response  Logic 

1  Go  to  E2  when  alternative  1  is  chosen. 

2  Go  to  E3  when  alternative  2  is  chosen. 

After -Frame  Logic 

1  Go  to  Tl0  if  all  of  the  following  are  correct: 
0M5.QM6.QM 7, QM8 


OPTIONS:  Frame  number,  d i sp ' ay/change  frame  logic 
NEXT,  d l sp 1  ay /change  next  frame  logic 
I,  insert  frame  logic  D,  delete  frame  logic 
BACK,  return  to  frame  list  page 
ENTER  choice  > 


Figure  19.  Authoring  Editor  Display  Showing  Completed  Response 
Contingent  and  After-frame  Branchino  Instructions. 
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Branching  Logic  for  Objective  2,  Frame  15 
_ Constructed  Response  Question  1 _ 

Pre-Frame  Logic 

1  Present  this  frame  if  all  of  the  following  are  not  presente< 

:  T4.T5 

2  Go  to  T 6  if  all  of  the  following  are  correct: 

QM3.QM4 

3  Go  to  T6  if  1  or  more  of  the  following  are  presented: 
t:.T3 


Response  Logic 

1  Go  to  El  when  alternative  l  is  chosen. 

2  Go  to  E2  when  alternative  2  is  chosen. 


flfter-Frame  Logic 

1  Go  to  E3  if  3  or  more  of  the  following  are  incorrect: 

C*H  .QM2.Qm.QM4.QCl 

2  Go  to  T t< . 


OPTIONS:  Frame  number,  disp’ay  change  frame  logic 

NEXT,  display  change  next  frame  logic 
I,  insert  frame  logic  D,  delete  frame  logic 
BMCK,  return  to  frame  list  page 
ENTER  choice  $> 


figure  c’O.  Authoring  Ulitor  I'isploy  Showing  Three  Classes 
of  Branching  Instructions. 


the  display  of  the  list  (shown  In  Figure  13)  of  all  of  the  frames  which 
have  been  defined  for  the  objective.  The  current  frame  (frame  number  24 
in  this  case)  is  Indicated  by  being  boxed  in.  The  author  Is  to  enter 
the  number  of  the  frame  to  which  the  branch  is  to  be  made  if  the  logic 
conditions  are  met--frame  number  28  in  this  example. 

Once  the  to-be-branched-to  frame  has  been  selected,  the  display 
changes  to  that  shown  in  Figure  14.  The  arrow  pointing  to  frame  number 
28  indicates  the  frame  to  which  the  branch  will  be  made  if  the  conditions 
are  met.  The  author  is  now  requested  to  select  the  class  of  conditions 
to  be  evaluated.  Assume  that  the  author  wishes  to  branch  if  some  number 
of  questions  was  answered  correctly  and,  therefore,  enters  "C."  The 
author  is  then  requested  (Figure  15)  to  enter  the  numbers  of  the  frames 
to  be  evaluated.  As  a  frame  number  is  entered,  an  asterisk  is  shown  to 
the  left  of  the  frame  in  the  list.  When  this  step  is  completed  (Figure 
lb),  the  author  is  asked  to  enter  the  minimum  number  of  questions  which 
must  be  answered  correctly  for  the  branch  to  be  taken--in  this  case,  all 
four.  This  completes  the  author's  entry  of  this  logic  instruction  and 
the  logic  defined  for  the  frame  thus  far  is  summarized  as  shown  in  Figure 
17.  At  this  point,  the  author  can  enter  another  logic  instruction,  go 
on  to  the  next  frame,  or  return  to  the  Frame  List  page. 

Assume  that  the  author  wishes  to  enter  a  Response  loqic  instruction. 
After  Options  I  (Insert  logic)  and  R  (for  Response  logic)  have  been 
selected,  the  text  of  the  question  is  displayed  with  the  correct 
answer(s)  indicated  to  refresh  the  author's  memory  as  to  its  content. 
Next,  the  author  is  shown  a  variation  of  the  Branching  Logic  Frame  List, 
as  illustrated  in  Figure  18.  For  each  alternative,  the  author  is  to 
choose  a  frame  number  from  the  list  or  simply  press  the  NEXT  key  if  no 
branch  is  to  be  taken  when  that  alternative  is  selected.  As  they  are 
entered,  the  numbers  of  the  alternatives  are  displayed  to  the  left  of 
the  corresponding  frames  in  the  list.  In  this  case,  the  author  has  in¬ 
dicated  a  branch  to  frame  E2  if  alternative  1  is  selected  and  is  in  the 
process  of  indicating  a  branch  to  Ed  if  alternative  2  is  selected.  After 
all  alternatives  have  been  considered,  the  logic  which  has  been  defined 
is  summarized  as  shown  in  Figure  19.  When  the  author  returns  to  the 
Frame  List  display,  the  existence  of  Response  and  After-frame  logic  will 
be  indicated  for  this  frame. 

Some  of  the  more  extensive  branching  possibilities  are  illustrated 
by  Figure  20. 

The  CAI  Presentation  Programs 

While  Authoring  Editor  provides  the  vehicle  by  which  LAI  materials 
are  developed,  there  must  also  be  software  to  support  presentation  of 
these  materials  to  students.  CAI  nresentation  is  accomplished  through 
the  use  of  a  general  program  structure  and  set  of  support  routines 
driven  by  the  CAI  module  description,  decision  logic,  and  text  records 
created  bv  an  author  using  the  Authoring  Editor.  Through  the  use  of 


chis  generalized  proqram  structure  and  a  table-driven  approach,  a  wide 
ranqe  of  computer-assisted  tutorial  and  drill  and  practice  instruction 
can  be  presented  with  minimal  programing  effort.  To  date,  three 
different  CAI  Presentation  Programs  have  been  developed  from  the  basic 
skeletal  structure  and  support  routines.  One,  the  first  developed,  pre¬ 
sents  standard  modules  in  support  of  lessons  assigned  on  the  student's 
first  pass  through  a  block.  The  second  supports  block  review  nodules, 
assigned  just  prior  to  a  student's  block  test,  which  present  material  for 
just  those  objectives  which  the  student  failed  while  studying  in  the 
block.  The  third  supports  block  remediation  modules,  assigned  after  a 
block  test  failure,  and  presents  material  for  just  those  objectives  that 
the  student  failed  on  the  test. 

Trie  basic  skeleton  and  support  routines  were  written  during  the 
development  of  the  first-pass  module  program.  This  program  required 
the  most  extensive  design  and  development  since  it  was  to  form  the 
basis  for  subsequent  programs.  The  block  review  and  remediation  pro¬ 
grams  were  then  created  by  slightly  modifying  the  main  loop  code. 
Development  time  for  variations  to  the  oasic  program  has  proven  to  be 
minimal  and  can  be  done  by  entry  level  progranners.  It  should  be  noted 
that  the  first-pass  module  presentation  program  is  sufficiently  general 
to  support  the  presentation  of  all  such  CAI  modules  written  for  any  AIS 
Course.  Similarly,  the  review  and  remediation  programs  are  general 
enough  to  nandle  the  presentation  of  any  cognitive  objectives  for  AIS 
block  review  and  remedia. ion. 

In  addition  to  supporting  CAI  presentation  for  student  study,  the 
same  presentation  programs  can  be  used  by  subject  matter  experts  and 
instructional  designers  who  wish  to  view  the  module  from  the  student's 
perspective. 

The  following  section  describes  the  typical  sequence  of  events  which 
occurs  wnen  a  student  interacts  with  a  CAI  module  of  the  type  developed; 
specifically,  assignment  to  a  first-pass  module.  The  subsequent  section 
describes  the  use  of  the  presentation  program  by  an  author  or  reviewer. 
The  student  performance  data  collection  actions  which  take  place  during 
the  presentation  of  a  CAI  module  are  mentioned  only  briefly.  A  full 
description  of  the  data  acquisition  and  analysis  process  is  presented 
in  the  subsequent  section. 

Student  CAI  Scenario.  As  was  discussed  in  the  Introduction  to  this 
report,  when  the  AIS  makes  a  lesson  assignment,  it  determines  whether 
there  are  two  or  more  alternative  modules,  including  any  CAI  modules, 
available  for  teaching  that  lesson.  If  so,  there  are  a  variety  of 
decision  rules  for  determining  which  module  should  be  assigned.  As  a 
general  rule,  CAI  modules  are  assigned  to  that  proportion  of  the  students 
for  whom  they  are  considered  to  be  the  most  appropriate,  assuming  that 
the  required  instructional  resources  (in  this  case,  an  interactive 
terminal)  are  available. 
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When  assigned  a  CAI  module,  the  student  signs  onto  any  available 
interactive  terminal  by  typing  in  his  or  her  student  account  number. 

If  the  AIS  does  not  recognize  the  number,  the  user  will  be  told  that  the 
account  number  is  not  registered  and  will  be  logged  off.  If  the  number 
is  recognized,  the  student's  Student  Data  Profile  record  Is  checked  to 
determine  whether  the  student's  current  assignment  is  indeed  a  CAI 
module.  If  not,  the  student  is  so  informed  and  logged  off. 

If  the  student  is  assigned  a  CAI  module,  the  particular  module  is 
determined  from  the  Student  Data  Profile  record.  The  Presentation  Pro¬ 
gram  then  reads  the  CAI  module  description  record,  created  by  the  Author¬ 
ing  Editor,  to  determine  whether  or  not  student  response  and  decision 
point  data  are  to  be  recorded  and  whether  student  comments  are  to  be 
elicited.  If  so,  data  recording  is  activated  for  those  classes  of  data 
which  are  to  be  collected. 

The  Presentation  Program  begins  by  presenting  the  material  in 
Objective  0,  containing  at  least  an  overview  of  the  lesson  and  a  required 
materials  list,  and  then  begins  the  instruction  contained  in  the  first 
numbered  objective  in  the  series.  Frame  descriptions,  branching  logic, 
and  text  records,  all  created  by  the  Authoring  Editor,  determine  the 
sequence  and  content  of  the  presentation.  While  authors  can  define  a 
variety  of  different  instructional  strategies,  a  single  approach  will  be 
described  here  for  explanatory  purposes.  First,  the  statement  of  the 
objective  is  presented  and  briefly  elaborated.  This  could  be  followed 
by  one  or  more  pages  of  text  and  a  set  of  practice  questions.  The  number 
of  practice  questions  could,  and  should,  be  a  function  of  the  student’s 
performance  on  prior  questions.  Additional  frames,  elaborating  specific 
problem  areas  will  be  presented,  as  necessary,  on  the  basis  of  tin* 
student's  responses.  A  typical  page  from  a  text  frame,  as  seen  by  the 
student,  is  presented  in  Figure  ^1. 

At  any  point  in  the  objective,  the  student  r.iay  opt  to  review  the 
material  that  has  already  been  presented  by  pressing  the  lUEh  key.  In 
review  mode,  text  is  displayed  in  the  normal  manner  and  questions  are 
displayed  with  the  student's  answers  indicat'd. 

Questions  and  the  feedback  and  prompts  following  incorrect  responses 
form  a  critical  part  of  trie  instructional  process.  For  both  constructed 
response  and  multiple  choice  questions,  students  are  required  to  continue 
answering  until  correct  or  until  reaching  the  specified  maximum  number 
of  attempts.  For  multiple  choice  questions,  the  student’s  last  response 
is  indicated  by  an  arrow  pointina  to  the  alternative  selected  while 
prior  responses  are  indicated  by  asterisks.  Figure  OJ  presents  an 
example  in  which  the  student  is  about  to  make  a  third  attempt  to  answer 
the  question,  ilote  the  author-supplied  feedback  (first  three  lines)  and 
prompt  (last  three  lines)  at  the  bottom  of  the  display.  On  constructed 
response  questions,  the  student's  prior  incorrect  responses  are  listed 
beneath  the  question  stem.  Figure  JJ  presents  an  example  in  which  the 
student  made  two  incorrect  responses  followed  by  the  right  answer,  nas 
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Objective  l  TOE  4  BACK  to  review 


fill  missiles  have  at  least  three  distinct 
sections.  Look  at  Figure  1.  The  Guidance  and 
Control  Unit,  sometimes  called  the  G  and  C  Unit, 
the  warhead  and  the  rocket  motor  are  the  component 
parts  of  a  missile. 

GUIDANCE  and  CONTROL  I  UtflRHERP  I  ROCKET  MOTOR 

t 

TYPICAL  MISSILE 
Ithree  sections^ 

Press  NEXT  to  go  on  or  C  to  comment  on  this  text. 


Figure  21.  Presentation  Proqram  Display  of  a 
Text  Frame. 
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Obi active  1  FRAHE  58 _ Press  BACK  to  review 


Detonation  of  the  RIM- 4  warhead  will  occur  when 

1.  either  the  contact  fuze  or  proximity  fuze 
contacts  the  target. 

*  2.  the  inpact  fu2e  contacts  the  target. 

-•  3.  the  missile  is  265  to  325  feet  from  the 

launching  aircraft. 

4.  the  triggering  areas  fire  broken  by  or  :ome 
in  contact  with  the  target. 


Tell  me  would  you  want  to  be  325  feet  from  an  AIiI-4 
issile  when  it  went  off1?  Neither  woui-4  a  pilot,  that 
ind  of  stuff  will  mess  up  your  hair. 

Okay ,  you 1  re  my  budc^<  so  Ill  give  you  a  hint. 

ITS  TtC  TRIGGREING  AREAS. 

Good  hint  huh1  I  got  all  the  answers. 

TRY  AGAIN 


F i yure  ZZ.  Presentation  Program  Display  of  Multiple  Choice 
Question  Showing  Feedback  and  Second  Attempt  Prompt. 


In  what  block  of  th«  flFTO  Form  35*  should  th« 
Federal  Supply  Classification  be  recorded? 

Enter  a  short  response  of  21  characters  or  less 
and  press  NEXT. 

1.  5 

2.  7 

3.  If 


GOT  IT.  Glad  you  finally  woke  up1 1 ' 
Press  7CXT  or  C  to  comment  on  this  question. 


Figure  23.  Presentation  Proqram  I'isplay  of  Constructed  Response 
Question  Answered  Correctly  on  tne  Third  Attempt. 


received  a  standard  feedback  message  for  a  correct  response  ami  is 
ready  to  continue. 


Over  tne  course  of  a  long  nodule,  the  author  nay  want  to  oncouraoe 
the  student  to  take  one  or  more  breaks;  to  exit  the  nodule  and  leave 
the  terninal  for  a  short  rest  period.  Other  forms  of  module  inter¬ 
ruptions  can  occur  due  to  computer  failure,  end  of  shift,  or  breaks  for 
.iieals.  In  each  case,  the  student  can  log  off  or,  if  the  Presentation 
Program  does  not  receive  a  keypress  for  a  specified  period  (currently 
lo  minutes),  it  is  assumed  that  the  student  has  left  the  terminal  and, 
after  displaying  an  inquiry  as  to  whether  anyone  is  there,  tne  student 
is  logged  off  automatically.  If  a  module  is  interrupted  for  any  reason, 
the  Presentation  Program  automatically  restarts  the  module  at  the  frame 
on  wnich  the  interruption  occurred  wnen  the  student  logs  back  onto  the 
terninal . 

After  a  period  of  instruction  and  practice,  an  objective  typically 
ends  with  a  series  of  test  questions.  Given  the  criterion  that  a 
certain  number  of  tne  questions  must  be  answered  correctly,  the  autnor- 
defined  brancning  would  normally  route  the  student  to  tne  end  of  the 
objective  as  soon  as  the  criterion  has  been  net.  If,  on  tne  other  nand, 
tne  student's  performance  is  below  criterion,  the  student  would  normally 
not  exit  the  objective  until  troublesome  points  have  been  reviewed  and 
retested  with  additional  test  items. 

Upon  exiting  the  objective,  the  stuoent  will  pass  through  a  frame 
aqainst  which  either  an  Objective  Passed  or  Objective  Failed  Flag  lias 
been  set.  If  the  objective  is  considered  to  be  prerequisite  to  a  sub¬ 
sequent  objective,  a  Lesson  Failed  Flag  would  normally  be  set  against 
the  same  frame  as  the  Objective  Failed  Flaq.  The  student  will  continue 
through  the  objectives,  in  sequence,  until  all  of  the  objectives  nave 
been  presented  or  until  the  Presentation  Program  encounters  a  Lesson 
Passed  or  Lesson  Failed  Flag. 

When  a  Lesson  Passed  or  Lesson  Failed  Flag  is  encountered,  the 
Presentation  Program  generates  a  module  test  form  containing  a  list  of 
any  objectives  failed  and  a  lesson  passed  or  lesson  failed  designator. 

The  Program  passes  this  form  to  the  main  AIS  management  program,  tne 
Adaptive  Model.  The  Adaptive  Model  records  the  student's  performance 
on  the  lesson  and  generates  the  student's  next  assignment.  This  assign¬ 
ment  is  displayed  on  the  terminal  for  the  student  to  copy.  The  student 
is  then  logged  off  the  terminal  and  the  module's  instructional  resource 
(the  terminal)  is  returned  to  the  resource  pool  for  reassignment.  The 
student  can  then  obtain  a  hard  copy  printout  of  the  next  assignment  by 
submitting  a  request  form  to  a  management  terminal. 

Author/Reviewer  Mode.  Access  to  the  CAI  Presentation  Program  is  not 
limited  to  students.  Lesson  authors  and  reviewers  may  wish  to  run  the 
Presentation  Program  to  verify  accuracy  of  module  content  and  to  view 
the  module  from  the  student's  perspective.  Author  and  reviewer  access 
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to  the  Presentation  Programs  is,  however,  handled  by  standard  AIS  pro¬ 
gram  access  methods  rather  than  being  under  the  control  of  the  Adaptive 
Model.  The  CAI  Presentation  Program  differentiates  between  students  and 
registered  authors  and  reviewers.  The  author/reviewer  does  not  have  a 
Student  Data  Profile  record  containing  a  current  assignment,  and  there¬ 
fore  is  required  to  enter  the  identifier  of  the  CAI  module  to  be  pre¬ 
sented. 


Having  accessed  a  particular  module,  the  author/ reviewer  may,  with 
a  few  exceptions,  take  the  module  like  a  student.  Unlike  a  student,  an 
author/reviewer  may  override  the  frame  control  logic  of  the  Presentation 
Program  and  request  the  presentation  of  any  frame  within  an  objective 
througn  the  use  of  a  special  function  key.  In  addition,  the  autnor/ 
reviewer  may  always  enter  cormients  about  the  material  that  is  being 
presented.  Student  comments  are  only  elicited  and  accepted  if  tne 
Stuuent  Connent  Module  Flag  lias  been  set  to  True.  Completion  of  a  module 
in  autiior/revicwer  mode  does  not  result  in  submission  of  a  lesson  com¬ 
pleted  or  failed  form  to  the  Adaptive  Model.  In  all  other  respects,  th * 
user's  interaction  with  the  Presentation  Program  is  identical  to  that  o 
a  student. 


Student  Performance  data  Acquisition  and  Analysis 

Tiie  CAI  data  Acquisition  and  Analysis  system  consists  of  four  major 
components:  data  recording  routines  in  the  Presentation  Programs;  a 
Data  Collection  Program  which  moves  student  performance  data  from  disk 
to  tape;  a  oata  Analysis  Report  Program  winch  generates  three  different 
types  of  reports;  and  a  Report  Submittal  Program  which  facilitates  users' 
requests  for  specific  reports. 

The  student  performance  data  collection  and  analysis  process  begins 
with  data  recording  routines  embedded  in  the  Presentation  Programs. 

These  routines  operate  interactively  with  data  being  recorded  directly 
onto  disk.  The  information  recorded  is  segregated  into  two  files: 
Response  Point  data,  which  are  recorded  following  every  frame;  and 
Decision  Point  data,  recorded  only  at  specified  Decision  Points. 
Periodically,  the  data  are  dumped  from  disk  to  tape  by  the  batch  CAI 
Data  Collection  Program.  The  CAI  Response  and  Decision  Point  history 
tapes  provide  the  primary  data  source  for  the  CAI  Data  Analysis  Report 
Program.  Tne  Report  Program  can,  nowever,  also  retrieve  recent  data 
stored  in  the  disk  files  for  applications  requiring  small,  current  stu¬ 
dent  samples.  There  are  both  backnround  (CAMIL)  and  batch  (PASCnL) 
versions  of  the  Data  Analysis  Report  Program.  The  former  is  used  to 
access  data  on  disk  while  the  latter  accesses  only  the  tape  data.  A 
single  CAM I L  program  provides  for  setup  and  submittal  of  both  the  back¬ 
ground  anu  batch  versions  of  tne  Report  Program. 

The  uata  collection  process  and  the  reports  available  are  discusseu 
in  greater  detail  in  the  following  subsections.  Although  tne  process 
employed  is  somewhat  different  than  for  performance  data  collection,  tne 
storage  and  retrieval  of  student  and  reviewer  comments  are  also  describee 
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udtj  Collection  and  Storage,  Whetner  student  perMr,  unco  data  aiu; 
com; lei.ls  arc  collected  during  the  presentation  of  a  C.»I  nodule  is  oeoen- 
uont  on  whether  the  modulo  author  nas  set  the  appropriate  data  collection 
flans  for  that  Module  via  the  Authoring  Lditor.  That  is,  for  data  within 
one  of  tne  three  categories  (“espouse,  decision  Point,  ana  Comments)  to 
bo  saved,  tne  author  uist  nave  set  the  data  Collection  Flat  for  Mat 
cateoory  to  True.  Tnis  philosoohv  of  limited  data  collection  was  adopter 
to  avoid  neneration  and  storaie  of  tne  intense  amounts  of  d.ata  uiicn 
would  otherwise  occur.  The  intent  is  Mat  data  be  collected  for  Mriative 
anti  suraative  evaluation  purposes  but  not  durino  normal  operations  exce  t 
for  consciously  initiated  sanplino. 

Kesponso  data  represents  the  nost  detailed  data  category.  The  data 
are  collected  at  the  end  of  each  frame  presented  to  the  student,  renard- 
less  of  frame  tyre,  and  include  the  followino  items: 

1.  Tlie  student's  Student  Account  Number. 

C.  The  frame  identifier. 

3.  Tne  number  of  the  student's  pass  tnrouon  this  frame. 

4.  Tne  current  date. 

j.  The  current  time  in  minutes  after  lidnicnt. 

i).  Tne  total  tine,  in  seconds,  spent  on  the  frame. 

7.  The  time,  in  seconds,  spent  in  review  mode  if  review 

was  initiated  from  this  frame. 

If  the  frame  is  a  question  frame,  the  following  data  are  also  collected: 

o.  The  number  of  attempts  made  to  answer  the  question. 

9.  The  number  of  the  alternative  selected,  bv  attempt  lumber 
(where  alternative  numbers  are  also  assioned  to  the  various 
anticipated  rec  onses  and  the  cateqorv  of  unanticipated 
response  foi  onstructed  response  questions). 

Id.  The  response  latency,  by  attempt  number. 

11.  The  number  of  unanticipated  responses. 

1C.  The  text  of  up  to  five  unanticipated  responses. 

Decision  Point  data  are  collected  at  the  end  of  the  modulo,  at  the 
end  of  each  objective,  and  at  the  end  of  each  frame  aqainst  which  a 
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Decision  Point  Flag  has  been  set.  The  data  collected  Include  the 
following: 

1.  The  student's  Student  Account  Number. 

2.  The  type  of  decision  point  (frame,  objective  or  module). 

3.  The  frame  Identifier. 

4.  The  number  of  the  student's  pass  through  this  frame. 

5.  The  current  date. 

6.  The  current  time  In  minutes  after  midnight. 

7.  The  elapsed  time  since  the  last  decision  point. 

3.  The  number  of  questions  presented  since  the  last  decision 
point. 

9.  The  number  of  questions  answered  correctly  since  the  last 
decision  point. 

10.  The  number  of  branching  logic  decisions  processed  for  the 
current  frame;  and  for  each  branching  logic  instruction 
processed: 

11.  The  branching  type  (Pre-frame,  Response  or  After- frame) . 

12.  The  number  of  the  Instruction  within  Its  type. 

13.  The  branch  actually  taken,  if  any. 

Both  Response  Point  and  Decision  Point  data  are  stored  on  disk  as 
they  are  collected.  Periodically  (e.g.,  once  a  week),  the  CAI  Data 
Collection  Program  is  run  to  delete  the  oldest  data  from  the  disk 
Response  and  Decision  Point  Files  and  transfer  the  data  to  the  CAI 
History  tapes. 

Collection  and  storage  of  student  (and  reviewer)  comments  is  handled 
somewhat  differently.  First,  of  course,  the  student  or  reviewer  must 
take  overt  action  to  enter  a  comment  against  a  frame  as  opposed  to  data 
collection  being  transparent  to  the  user.  Up  to  eight  comments  are 
stored  for  each  frame  in  a  "circular"  file  on  disk.  That  is,  when  a 
ninth  comment  Is  entered  against  that  frame,  its  content  replaces  that 
of  the  first  (oldest)  comment.  The  content  of  the  Comment  files  Is 
never  transferred  to  tape. 

Oata  Analysis  Reports.  There  are  four  different  CAI  Data  Analysis 
Reports  available  to  authors  and  evaluators:  the  Decision  Point  Data 


Report,  the  Response  Analysis  Data  Report,  the  Unanticipated  Response 
Report,  and  the  Comnents  Listing.  In  addition,  standard  A1S  CM  I  reports 
can  be  used  to  provide  a  description  of  overall  module  performance. 

All  reports  are  requested  from  an  interactive  terminal.  For  the 
three  student  performance  reports,  requests  are  submitted  via  a  TAMIL 
program  (CA1  Reports  Proqram)  which  prompts  the  user  to  enter  the  various 
report  request  parameters.  Thus,  the  user  does  not  need  to  learn  how  to 
set  up  job  control  and  input  data  cards.  The  report  request  parameters 
include: 

I.  The  module  identifier  (Course,  Course  Version,  block. 

Lesson  and  Module  numbers). 

L’.  The  type  of  report. 

J.  The  date  constraints  (the  period  from  which  data  are  to 
tie  drawn). 

4.  The  input  medium  (disk  or  tape). 

‘j.  The  objectives  within  the  module  which  are  to  be  reported. 

u.  If  desired  by  the  user,  the  numbers  of  the  specific  frames 
within  each  objective  to  be  reported  (where  a  set  of 
frames  is  defined  by  the  numbers  of  the  first  and  last 
frames  in  the  set). 

Once  the  request  parameters  have  been  defined,  the  report  submittal 
program  sets  up  the  necessary  tiles,  submits  the  job  to  generate  the 
r.  port  on  the  central  line  printer,  and  gives  the  user  the  job  number 
which  will  appear  on  the  report  printout. 

The  Decision  Point  Report  is  generated  from  data,  stored  either  on 
disk  or  tape,  in  the  Decision  Point  Data  File.  This  report  provides  a 
summary  of  student  performance  within  each  objective  and  within  those 
intraobjective  segments  (sets  of  frames)  which  the  author  has  defined 
by  setting  Decision  Point  Flags  at  the  beginning  and  end  of  each  segment. 
An  example  page  from  the  Decision  Point  Report  is  presented  in  figure 
J4.  t ach  component  of  the  Report  contains  the  number  and  name  of  the 
frame  representing  that  Decision  Point;  whether  the  data  reported  per¬ 
tains  to  students'  first,  second,  or  subsequent  passes  through  that 
point;  the  elapsed  time,  number  of  questions  answered  and  number  answered 
correctly  since  the  last  Decision  Point;  the  branching  logic  evaluated 
at  this  point;  and  the  number  and  percentage  of  students  taking  each 
branch . 

An  example  page  from  the  Response  Analysis  Report,  is  presented  m 
Figure  JO.  Lach  component  of  the  report  is  identified  bv  tne  number  and 
name  of  the  relevant  frame  and  whether  the  data  reported  pertains  to 
students'  first,  second,  etc.,  pass  through  this  frame,  lor  frames 
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Figure  25.  Example  Page  from  Response  Analysis  Report. 


other  than  question  frames,  only  time  data  are  reported.  For  question 
frames,  a  matrix  format  is  used  to  present  student  performance  and  re¬ 
sponse  latency  data  as  a  function  of  the  response  (multiple  choice  alter¬ 
native  or  constructed  response)  selected  on  successive  attempts.  The 
margins  of  the  matrix  provide  a  sunmary  of  student  performance  on  the 
question  (total  percentage  correct,  percentage  correct  by  attempt,  and 
total  time  to  correct  response!  while  the  matrix  cells  provide  a  more 
detailed  picture  of  how  students  reacted  to  the  question. 

An  example  page  from  the  Unanticipated  Response  Report  is  presented 
in  figure  2 t>.  For  each  constructed  response  question,  each  unique  un¬ 
anticipated  response  Is  listed  in  order  of  frequency  of  occurrence 
together  with  the  number  of  times  which  that  particular  response  has 
been  entered. 

Comments  Listings  are  requested  via  the  Authoring  Editor  rather  than 
the  CAI  Reports  Program,  frames  against  which  comments  have  been  made 
are  indicated  on  the  Editor's  frame  List  display  for  each  objective.  The 
user  can  request  that  comments  made  on  a  particular  frame  be  displayed 
at  the  terminal  or  that  comments  on  one  or  more  frames  be  listed  on  the 
central  line  printer.  The  user  also  nas  tne  option  of  having  the 
consents  ('urged  from  the  file  as  they  are  displayed  or  listed. 

CAI  Materia  1  Print  Program 

As  authors  create,  review,  and  revise  CAI  modules,  it  is  often  use¬ 
ful  to  work  from  a  hard  copy  printout  of  the  module's  content  in  addition 
to,  or  in  place  of,  the  displays  provided  by  the  Authoring  Editor.  There 
are  also  instances  in  which  hard  copy  printout  is  desirable  for  student 
use.  A  feature  of  the  Authoring  Editor  Is  the  capability  to  request  a 
variety  of  printed  listings  of  CAI  module's  content  and  branching  logic. 
The  Editor  queries  the  author  for  the  desired  print  options  and  then 
Initiates  a  special  background  (non-lnteractive,  low  priority)  program 
to  produce  the  printouts. 

Author  Listings.  In  addition  to  listings  of  student  comments,  four 
different  types  or  printer  listings  are  available  to  authors,  ranging 
from  sunmary  information  to  detailed  listings  of  frame  contents.  The 
option  of  requesting  multiple  copies  is  available  for  each  type  of 
listing. 

At  the  most  general  level,  the  Module  Surinary  Listing  provides  an 
overview  of  all  of  the  CAI  modules,  operational  or  under  development, 
currently  defined  in  the  data  base.  The  information  provided  for  each 
module  Includes  the  module  Identifier  (Course  Number  throuqh  Module 
Number),  module  title,  author's  10  (Social  Security  Number),  and  the 
number  of  objectives  defined  within  the  module. 

For  a  particular  module,  the  Frame  Sunmary  Listing  provides  an  over¬ 
view  of  the  content  of  individual  objectives.  An  example  of  this  listing 


Figure  26.  Example  Page  from  Unanticipated  Response  Report. 


is  presented  in  Fiqure  27.  The  information  provided  by  this  listing  is 
essentially  the  same  as  the  Editor's  Frame  List  Display  .  Each  frame  in 
the  objective  is  listed  by  number  and  frame  name.  The  existence  of  any 
branching  logic  and  Frame  Flags  is  noted  and  the  maximum  number  of 
attempts  allowed  to  answer  questions  are  shown.  If  a  frame  is  an  alias 
(i.e.,  references  another  frame),  the  referenced  frame  is  identified. 

Tne  most  frequently  used  printout  is  probably  tne  Frame  Contents 
Listing:  the  complete  printout,  by  frame,  of  all  text  and  question 
material.  Such  a  listing  can  be  requester  for  an  entire  nodule,  an 
individual  objective,  a  specified  set  of  frames  or  a  single  frame.  Tne 
material  contained  in  a  textual  content  frame,  up  to  the  full  four  pages, 
is  printed  on  a  single  printer  page  with  appropriate  headings.  An 
example  printout  of  a  Text  frame  is  shown  in  Fiqure  23.  For  a  Question 
frame,  the  printout  includes  the  question  stem,  the  alternatives  or  anti¬ 
cipated  responses  with  the  correct  answers  denoted,  and  all  author- 
supplied  feedback  and  prompt  messages.  An  example  is  presented  in 
Figure  29. 

Finally,  the  Branching  Logic  Listing,  an  example  of  which  is  snown 
in  Figure  30,  provides  a  nard  copy  listing  of  all  of  the  branching  logic 
wnich  has  been  defined  for  frames  within  an  objective.  The  format  in 
which  information  is  presented  is  similar  to  the  Editor's  Branching 
Logic  displays. 

Printed  Materials  for  Student  Use.  In  addition  to  the  various 
author's  listings,  hard  ropy  printout  of  a  CAI  module's  content  can  be 
requested  in  a  format  appropriate  for  direct  use  by  students  as  a 
programmed  text.  Special  purpose  (Documentation  and  Branching  Decision) 
frames  are  automatically  suppressed,  and  the  author  can  elect  to  suppress 
any  otiier  frames  by  setting  "Skip  this  Frame"  flags  via  the  Editor.  All 
other  frames  are  printed  in  the  order  in  which  they  occur  in  the  module. 
Branching  logic  is  simply  ignored. 

For  textual  content  frames,  one  screen  display  (a  frame  page)  is 
printed  on  eacn  page.  For  question  frames,  each  question  is  identified 
by  a  number,  assigned  sequentially.  For  constructed  response  questions, 
only  the  question  stem  is  printed,  while  the  alternatives  and  their 
numbers  are  printed  for  multiple  choice  questions.  The  answers  to  all 
questions,  identified  by  number,  are  listed  on  a  test  key  page  following 
the  body  of  the  module.  Page  numbering  is  provided  automatically. 

Material  is  currently  printed  in  a  double-spaced  format.  Boxes 
which  the  author  has  used  to  emphasize  material  in  the  module  are  shown, 
as  are  any  of  the  simple  graphics  which  the  author  may  have  provided. 

For  such  graphics,  a  symbol  is  substituted  for  the  solid,  "unrecog¬ 
nized  character"  symbol  typically  employed.  As  an  option,  the  author 
may  request  that  the  material  be  printed  on  special  unlined,  3  1/2"  by 
11"  paper.  The  option  of  printing  multiple  copies  is  also  available. 


f»**e  s*j««a**  listing  i  lesson  module  > 
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Figure  27.  Example  Frame  Surrrnary  Listing  Printout. 


Frame  Contents  Showing  a  Question  Frame. 


There  are  a  variety  of  uses  for  such  printouts.  They  can  be  used 
as  hard  copy  backups  for  students  assigned  CAI  modules  in  the  case  of 
computer  failure.  They  are  useful  to  instructors  tor  answering  the 
questions  of  students  assigned  CAI  nodules.  Their  most  important 
function  however,  may  well  be  as  a  first  step  toward  the  on-line  develop¬ 
ment,  evaluation,  and  revision  of  materials  intended  for  off-line  use. 

Authoring  System  Documentation 

A  CAI  Authoring  Editor  User's  Manual  was  written  to  describe  the 
CAI  concepts  supported  by  the  Editor,  the  mechanics  of  its  use,  the  CAI 
Presentation  Programs,  the  various  CAI  Reports  available  and  the  pro¬ 
cedures  by  which  they  are  requested,  and  the  availability  and  use  of  the 
different  types  of  author  and  student  printouts.  The  Manual  was  actually 
developed  on-line  as  a  CAI  module  (without  branching  or  question  frames) 
and  is  available  to  novice  authors  via  any  of  the  AIS  interactive  ter¬ 
minals.  Hard  copy  versions  of  the  manual  are  also  produced  as  needed 
by  the  CAI  Material  Print  Program. 

This  approach  has  assured  rapid  and  easy  revision  of  the  manual  to 
provide  up-to~date  documentation  as  the  Editor  and  its  various  support¬ 
ing  programs  are  expanded  and  refined.  Under  a  companion.  Authoring 
Procedures  contract,  the  Manual  is  being  expanded  to  include  information 
on  the  selection  of  content  for  CAI  treatment,  instructional  strategies 
appropriate  to  CAI,  and  guidelines  for  evaluation  and  revision  of  CAI 
nodules. 

Although  it  had  not  been  implemented  at  the  time  this  report  was 
prepared,  it  is  intended  that  a  second  type  of  Editor  documentation  will 
be  provided.  "Help"  routines  will  be  imbedded  in  the  Editor  itself. 

That  is,  through  the  use  of  a  special  Function  Key,  the  author  will  be 
able  to  access  information  pertaining  to  the  use  of  the  Editor  Options 
which  are  available  to  him/her  from  the  currently  displayed  page  of  the 
Editor.  These  Help  routines  will  access  the  information  contained  in 
appropraite  frames  of  the  User's  Manual.  Thus,  mo st  required  revisions 
to  the  documentation  will  only  need  to  be  made  in  one  place,  in  the 
manual  itself.  The  Help  sequences  will  then  access  the  updated  infor¬ 
mation  automatically. 

documentation  to  support  subsequent  maintenance  and  revisions  of 
the  Editor  itself,  the  Presentation  Programs  and  the  various  supporting 
programs  is  provided  by  a  Part  II,  Product  development  Specification. 

This  Part  II  Spec  was  also  produced  on-line  (via  a  different,  less 
structured  Editor),  is  stored  on  tape  and  is  accessible  for  the  pro¬ 
duction  of  hard  copy  documentation. 


IV  WLLMENTATION  AMO  L VALUATION 

In  addition  to  the  CaI  support  software  effort  described  nere,  tne 
CAI  development  project  also  contained  a  second  authoring  procedures 
effort.  The  purposes  of  this  second  effort  were  to  (a)  uefine  and  docu¬ 
ment  a  set  of  procedures  for  CAI  materials  selection,  prouuction,  eval¬ 
uation,  and  implementation  in  the  AIS  environment,  (b)  evaluate  the 
utility  of  the  total  CAI  authoring  system  by  developing,  implementing 
ana  evaluating  a  number  of  CAI  modules  in  an  AIS  course,  and  (c)  train  a 
small  number  of  ATC  personnel  in  the  use  of  the  procedures  and  support 
software.  The  results  of  this  effort  are  reported  in  full  in  Part  II  of 
tiiis  report  (Lewis,  Lovelace,  Mahany  and  Judd,  in  press)  and  will  be 
outlined  only  briefly  here. 

The  authoring  procedures  effort  began  with  a  rough  definition  of 
the  procedural  model  which  was  then  revised  and  refined  on  the  basis  of 
experience.  Following  the  steps  outlined  by  the  procedural  model,  work 
began  by  selecting  candidate  lessons  for  the  CAI  ueiuonstration.  Of  toe 
four  AIS  courses,  tne  Weapons  Mechanic  course  was  considered  tne  most 
promising  because  of  its  combination  of  varied  subject  matter,  large 
student  flow,  and  recent  relatively  ooor  field  performance  data. 

Standard  AIS  till  reports  were  used  to  identify  six  lessons  in  two  olocks 
of  tne  course  wnicn  demonstrateu  unacceptably  nigh  lesson  test  failure 
rates,  relatively  nigh  failure  rates  on  the  enu-of-block  tests  for  tne 
objectives  contained  in  these  lessons,  and  wnich  contained  subject  matter 
unlikely  to  be  modified  in  the  near  future.  These  lessons  were  targeted 
for  tnree  types  of  CAI  application:  (a)  six  modules  to  be  used  as  alter¬ 
native  treatments  on  students'  first  pass  tnrough  the  block,  (b)  two 
block  review  modules  containing  CAI  treatment  of  the  objectives  from 
these  lessons,  and  (c)  two  block  remediation  modules,  again  containing 
CAI  treatment  of  the  objectives  from  tne  six  lessons. 

After  examining  the  existing  materials  and  tests,  it  was  concluded 
that  neither  the  lesson  tests  nor  those  portions  of  the  block  tests  per¬ 
taining  to  the  relevant  objectives  were  adequately  valid  or  reliable  for 
evaluating  the  effectiveness  of  the  CAI  treatments.  Consequently,  both 
forms  of  the  two  block  tests  and  two  forms  of  each  of  the  six  lesson 
tests  were  substantially  revised  and  expanded  to  more  closely  match  the 
stated  performance  requirements  of  the  Specialty  Training  Standard.  Once 
the  revised  tests  had  been  approved  by  course  personnel,  they  were  imple¬ 
mented  in  paper-and-penci 1  form  for  all  students  in  the  course. 

Work  then  began  on  developing  six  CAI  modules  for  administration 
during  students'  first  pass  through  the  block.  While  existing  materials 
were  available  in  the  form  of  programmed  text  and,  in  some  cases,  audio¬ 
visual  modules,  the  content  was  substantially  revised  to  more  closely 
match  the  requirements  of  the  Specialty  Training  Standard.  Instructional 
strategies  were  also,  of  course,  tailored  for  CAI  presentation. 

Work  started  on  three  of  the  modules  before  the  Authoring  Editor 


7o 


was  ready,  but  after  it  had  been  desiqned  and  the  module  structure  de¬ 
fined.  Tnerefore,  the  earliest  authoring  was  done  on  paper  display 
forms.  When  a  rudimentary  form  of  the  Editor  became  available,  text  and 
questions  which  had  been  prepared  on  the  forms  were  input  by  a  secretary. 
It  was  thought  that  some  of  the  authors  might  prefer  the  use  of  forms  and 
wish  to  continue  with  this  approach  but,  as  additional  Editor  features 
became  available,  all  of  the  authors  found  it  more  convenient  to  input 
and  format  the  materials  themselves.  All  of  the  work  on  the  last  three 
lessons  was  accomplished  this  way.  Surprisingly  few  problems  were  en¬ 
countered  during  module  development,  but  many  aspects  of  the  Authoring 
Editor  were  shaped  by  frequent  interactions  between  tne  authoring  and 
software  teams. 

As  soon  as  the  modules  were  completed  and  their  content  revieweu 
by  Weapons  Mechanic  course  instructors,  they  were  tried  out  on  a  one-on- 
one  basis  with  a  small  number  of  student  volunteers.  A  number  of  minor 
revisions  were  made  on  the  basis  of  tnese  reviews  and  tryouts,  and  tne 
modules  were  implemented  in  the  course  for  purposes  of  formative  evalu¬ 
ation.  Following  revisions  made  on  the  basis  of  tnis  evaluation,  the 
modules  were  reimplementeu  for  sui. native  evaluation.  The  results  of  this 
evaluation  are  presented  in  Part  II  of  this  report.  Tne  objectives  from 
the  first-pass  modules  were  then  copied,  revised  ana  shortened,  and  com¬ 
bined  to  form  two  block  review  modules  and  two  block  remediation 
modules . 

Tne  CmI  authoring  team  consisted  of  tnree  members.  All  were  exper¬ 
ienced  tecnnical  training  authors  (of  programmed  text  and  audio-visual 
materials)  but  none  had  any  prior  CAI  authoring  experience.  In  fact, 
none  nad  even  used  CAI  as  a  stuuent.  Only  one  of  the  three  could  be 
considered  a  Weapons  Mechanic  subject  matter  expert. 

As  is  often  the  case,  the  team  did  not  keep  accurate  records  of 
developiient  times  but  times  can  be  estimated  for  the  six  first-pass 
modules.  At  the  end  of  the  first  b  months  of  the  project,  tne  first-pass 
nodules  nad  been  revised  and  implemented  for  summative  evaluation.  The 
team  leader  spent  relatively  little  time  actually  authoring,  concentrat¬ 
ing  insteau  on  defining  the  procedural  model,  producing  the  CAI  Author's 
Handbook,  interfacing  with  the  software  personnel,  and  attending  to 
administrati ve  problems.  T he  other  two  team  members  were  occasionally 
called  upon  for  assistance  on  other  on-going  projects.  A  liberal  esti¬ 
mate  of  tne  total  time  spent  in  developing,  evaluating,  and  revising  tne 
six  modules  is  JcJu  manhours.  Tnis  includes  time  spent  in  revising  the 
uIock  and  lesson  tests  even  tnouoh  block  test  revision  was  technically 
not  part  of  tne  CmI  effort. 

According  to  tne  course's  Plan  of  Instruction  \P0I),  tne  content 
taught  by  the  cAI  modules  was  equivalent  to  approximately  Co  classroo 
hours.  On  this  basis,  CAI  development  required  do  mannours  per  PA  I  hour. 
Average  student  completion  time,  totalled  across  tne  six  modules,  was 
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1...7  nours.  Tnis  results  in  an  estinated  IK.  manhours  per  student 
contact  tiour.  These  values  are  comparable  to  development  times  for 
prograf.i.ied  text  materials  in  this  environment. 

Usability  of  tne  Authorin']  Editor  was  also  evaluated  tnrough  train¬ 
in')  tnree  ATC  instructors  in  use  of  the  editor.  These  instructors  were 
drawn  from  three  different  courses  supported  by  the  AIS-- Inventory 
Manuqei lent ,  Precision  Measuring  Equipment,  and  Weapons  Mechanic,  done 
of  the  author  trainees  were  computer  programmers  and  none  nad  prior  CAI 
development  experience.  The  training  period  lasted  for  3  weeks,  during 
wnicn  the  trainees  came  to  the  contractor's  facility  for  3  to  4  tiours 
each  morning.  The  first  session  was  spent  in  providin']  an  overview  of 
the  AIS  CAI  system  and  the  role  of  CAI  within  the  AIS  and  in  introducing 
tiie  trainees  to  the  Authoring  Editor  and  tne  CAI  Author's  Handbook.  Ho 
formal  training  took  place  during  the  subsequent  14  sessions.  Using  the 
Handbook  as  a  reference  manual,  each  of  the  trainees  used  tne  Authoring 
Euitor  to  develop  a  CAI  module  in  the  area  of  their  own  specialty.  Con¬ 
tractor  personnel  were  available  to  answer  questions  and  to  review  and 
comment  on  the  trainees'  work.  In  many  cases,  author  trainees  were  also 
able  to  work  on  the i r  modules  durinq  the  afternoons  while  performing 
their  normal  classroom  duties. 

The  author  trainees  asked  relatively  few  questions  after  the  first 
few  sessions,  ilost  of  the  suggestions  made  by  contractor  personnel  per¬ 
tained  to  tne  need  for  more  frequent  questions  in  the  modules  and  in¬ 
creased  individual ization  through  branching.  At  the  end  of  the  3  week 
period,  each  trainee  nad  devr’uped  a  module,  had  nad  it  reviewed  by  the 
contractor  and  other  ATC  personnel,  had  run  single  student  tryouts,  and 
nad  made  minor  revisions  (*•;  tne  basis  of  these  reviews  and  tryouts.  Tne 
consensus  of  tnose  reviewing  the  modules  was  that  they  were  generally  of 
good  quality  and  nau  capitalized  fairly  well  on  the  capabilities  of  CAI. 
One  of  the  modules  was  subsequently  implemented  in  the  Weapons  Mechanic 
course.  The  time  expended  by  the  author  trainees  on  this  first  module, 
through  revision  following  single  student  tryouts,  was  approximately  40 
nours  per  student  contact  hour. 


V  CONCLUSIONS  AND  RECOMMENDATIONS 


The  Authoring  Editor  approach  to  facilitating  CAI  development 
appears,  at  this  time,  to  be  very  promising.  Experience  to  date,  while 
admittedly  limited,  has  demonstrated  that  reasonably  effective  CAI 
modules  can  be  produced  at  a  very  acceptable  cost  in  terms  of  manhours 
per  hour  by  personnel  without  prior  CAI  authoring  experience.  In 
addition,  it  has  been  shown  that  ATC  personnel  can  learn  to  use  the 
Editor  in  a  reasonable  period  of  time  without  formal  training.  While  it 
had  been  anticipated  that  the  ATC  author  trainees  would  have  numerous 
suggestions  regarding  desired  changes  to  the  Authoring  Editor,  this  was 
not  found  to  be  the  case.  They  were,  in  general,  quite  satisfied  with 
the  Editor  and  all  expressed  an  interest  in  having  CAI  implemented  in 
their  respective  courses.  As  it  stands,  the  authoring  system  appears 
ready  for  use  by  ATC  instructional  development  and  evaluation  personnel. 

In  assessing  the  various  features  and  components  of  the  authoring 
system,  the  major  contributor  to  simplifying  the  task  and  hence  reducing 
costs  is  probably  elimination  of  any  need  for  the  author  to  work  in  a 
computer  language.  All  of  the  programming  work  has  been  done  beforehand 
and  provided  in  the  form  of  the  Editor  and  Presentation  Programs.  Future 
needs  for  programming  effort  will  depend  on  how  adequately  this  software 
meets  the  requirements  of  future  applications.  Although  it  certainly 
cannot  be  proven,  the  authors  of  this  report  think  that,  due  to  the 
flexibility  built  into  it,  the  existing  software  could  serve  the  needs 
of  the  AIS  environment  for  some  time  to  coir:.  Eventually,  however,  it 
is  anticipated  that  developing  author  expertise  will  justify  increased 
software  capability. 

The  second  greatest  contributor  to  facilitating  the  author's  task 
is  probably  the  extent  to  which  the  task  is  structured  by  the  Editor. 

The  overall  structure  of  the  module  is  determined  for  the  author,  units 
within  this  structure  are  matched  to  the  requirements  of  the  environment, 
and  the  occurrence  of  critical  units  is  either  forced  or  prompted. 

While  the  author  retains  a  great  deal  of  flexibility,  this  flexibility 
is  exercised  through  selection  of  specific  options  which  provide  a 
degree  of  control  over  the  authoring  process,  while  reminding  the  author 
of  the  various  courses  of  action  which  may  oe  taken. 

A  third  major  factor  in  facilitating  authoring  is  undoubtedly  the 
numan-engineered,  computer-aided  input,  formatting,  and  editing 
capability  provided  by  tne  Editor.  Other  than  tne  approach  to  defining 
branching  loqic,  there  is  little  here  that  is  radical  or  even  novel.  The 
work  involved  only  the  application  of  existing  technology  to  a  particular 
problem  area.  Given  the  diminishing  cost  of  computer  use,  there  is 
little  reason  not  to  provide  authors  with  the  benefits  of  this 
?  hnul  ogy . 


•«t  this  time,  it  is  difficult  to  evaluate  the  utility  of  the  auto 
.  structured  student  nerformance  data  collection  and  analysis 


routines.  Toe  reports  die  not  become  available  until  the  later  stages 
of  formative  evaluation  of  the  six  first-pass  modules.  During  their 
limited  period  of  use,  they  did  appear  to  be  easy  to  use  and  interpret. 
Tne  members  of  this  particular  authorin']  team  were,  however,  accustomed 
to  collection  such  data  and  using  the  data  to  improve  instructional 
materials,  for  broader  application,  more  extensive  prompting  and 
guidance  in  use  of  tne  data  collection  routines  might  be  desirable. 

It  suould  be  noted  tnat  the  software  development  effort  reported 
here  was  substantial.  Given  the  extent  of  tne  various  functions  to  lie 
supported,  the  authors  of  tiiis  report  believe  that  it  would  not  nave 
been  possible  to  complete  the  project  within  the  temporal  aau  fiscal  con 
straints  of  toe  contract  if  it  had  not  been  for  the  availability  of  the 
IAMIL  language,  the  software  development  capability  of  the  Alb,  and  the 
wel 1 -structured  nature  of  the  CM I  system  in  which  the  CAI  was  embedded. 
The  relative  ease  with  which  CAM1L  code  could  be  produced  and  debugged 
allowed  the  developers  to  experiment  witn  a  number  of  different 
approaches  to  various  problems,  obtain  user  feedback  on  these  different 
approaches,  and  revise  the  code  accordingly.  Tiie  process  was  essential! 
that  of  formative  evaluation  and  revision  within  the  domain  of  human- 
engineered  software.  These  same  characteristics  of  CAM 1 L  and  the  A I S 
will  facilitate  any  future  expansion  of  the  C A 1  authoring  system. 

With  respect  to  recoixiendations  for  future  activities,  the  most 
critical  action  that  must  be  taken  if  the  effort  reported  nere  is  to  be 
justified,  is  that  the  C  A I  authorin')  system  be  used.  The  results  of  the 
evaluation  described  in  Part  II  of  th i s  report,  although  limited,  strong 
ly  indicate  that,  used  judiciously,  C A 1  can  serve  a  useful  function  in 
a  technical  training  environment  such  as  that  of  the  AIS.  i urther,  usa¬ 
bility  evaluation  results  reported  both  nere  and  in  Part  II  demonstrate 
that  CAI  development  can  be  made  cost  effective  through  use  of  the 
authoring  system. 

It  is  recognizee  that  there  are  certain  inadequacies  in  the  author¬ 
ing  system,  primarily  in  the  areas  of  response  processing  and  autnor- 
generated  graphics.  These  areas  provide  two  of  the  most  likely 
candidates  for  future  software  development. 

As  was  discussed  in  Section  II  of  this  report,  answer  processing 
for  constructed  response  questions  has  been  a  problem  area  in  CAI  devel¬ 
oped  for  military  technical  training.  While  sophisticated  algorithms 
exist  to  aid  in  response  recognition,  the  C A I  systems  employing  these 
algorithms  have  not  made  the  authoring  process  sufficiently  simple  for 
the  algorithms  to  be  used  correctly.  It  is  recoiamended,  therefore,  that 
a  response  processing  dialog  be  developeu  and  added  to  the  Authoring 
Lditor  which  would  guide  the  author  in  defining  anticipated  responses  to 
constructed  response  questions.  The  encoding  algorithm  employee  by 
PLATO  could  be  used  to  recognize  misspellings.  As  currently  conceived, 
this  dialog  would  first  prompt  the  author  to  enter  an  anticipated  re¬ 
sponse.  The  Lditor  would  break  the  response  down  into  its  component 
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words  and  ask  the  author  to  identify  the  "important"  words  in  the 
response.  Noise  words  such  as  articles,  prepositions,  and  auxiliary 
verbs  would  be  recognized  and  the  author  would  be  questioned  as  to  their 
importance.  Next,  the  author  would  be  asked  if  word  order  is  important 
anu,  if  so,  would  be  prompted  to  indicate  important  word  groups  and  any 
ordering  restrictions  within  and  among  word  groups,  finally,  the  author 
would  be  prompted  to  enter  synonyms  for  the  important  words.  An  approach 
such  as  tiiis  would  not  only  simplify  the  authoring  process  but  would 
help  the  author  to  identify  the  critical  aspects  of  response  judging. 
This,  in  turn,  should  promote  generation  of  better  anticipated  responses 
and  constructed  response  questions  of  more  uniform  qjality. 

Tne  software  described  in  this  report  makes  no  provision  for  author 
generation  of  graphic  displays.  Although  sucii  displays  can  be  construc¬ 
ted  through  use  of  the  CAMIL  language,  it  is  recommended  that  a  graphics 
editor  be  developed  which  would  allow  the  non-programming  author  to 
define  drawings  in  terms  of  basic  geometric  elements  (e.g.,  straight 
lines,  circles,  and  arcs).  Such  an  editor  would  be  relatively  efficient 
in  terms  of  storage  and  redisplay  since  only  tne  basic  elements  of  tne 
drawing  need  to  be  stored  and  should  prove  adequate  for  preparing  simple 
figures  and  almost  all  schematic  diagrams. 

If  more  complex  drawings  are  required,  nowever,  it  becomes  much 
more  difficult  for  the  author  to  define  the  figures  in  terms  of  geometric 
shapes.  While  it  is  possible  to  define  "freehand"  shapes  through  key¬ 
board  control  over  a  cursor,  the  process  is  substantial ly  simplified 
through  use  of  a  light  gen  or  digitizing  device.  Iho  plasma  display 
terminals  used  by  the  AIS  cannot  support  light  pen  capability.  Two 
forms  of  digitizers  are  avai lable--video  and  tablet.  A  video  digitizer 
uses  a  video  camera  to  scan  the  drawing  and  convert  it  into  a  dot  matrix. 
To  use  a  tablet  digitizer,  the  author  overlays  a  special  electronic 
tablet  with  the  drawing  to  be  reproduced  and  then  uses  a  stylus  to  trace 
tnat  portion  of  tne  drawing  which  is  to  be  transmitted  to  the  host 
computer  and  stored.  Typically,  the  tracing  process  can  be  either  con¬ 
tinuous  or  stepwise.  The  major  problem  wito  a  video  digitizer  or 
continuous  tracing  on  a  tablet  digitizer  is  tnat  tne  drawing  is  repre¬ 
sented  in  terms  of  dots  rattier  than  ueometric  elements.  Redisplay  of 
single  dots  on  a  vector  terminal  is  an  extremely  time-consuming  process. 

I ne  digitized  matrix  is  really  suitable  onlv  for  redisplay  on  a  terminal 
with  a  refresh  memory  winch  can  he  preloaded  with  tne  graphic,  tor  a 
graphics  digitizer  to  be  feasible  for  graphics  generation  using  the 
current  AIS  terminals,  software  would  ho  reguireu  to  transform  tne  dot 
matrix  into  a  number  of  basic  geometric  elements  which  could  he  more 
guioklv  redisplayed.  Such  software  would  require  contour  recondition 
routines  wnicii  would  he  non-trivial  to  develop. 

It  is  recornendod  that  the  use  of  a  tablet  digitizer  connected 
directly  to  a  graphics  editor  be  invest  Mateo.  With  such  an  ap'ro.u.'i, 
the  stylus  of  the  digitizer  could  he  used  like  a  hunt  pen,  and  the 
tablet  could  contain  a  menu  of  the  uroietru  shapes  recoumzod  b>  me 
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editor.  To  use  such  a  system,  tno  author  would  select  the  qeometrm 
element  to  be  reproduced  from  the  menu  and  note  the  points  representin', 
the  limits  of  toe  element,  for  example,  after  the  snape  was  defined 
from  the  menu,  a  line  segment  would  be  entered  as  the  two  end  points  of 
the  line;  a  circle  would  be  entered  as  the  center  point  and  a  point  on 
the  circumference.  This  approach  could  combine  the  strono  points  of 
both  the  digitizer  and  a  graphics  editor. 

In  a  totally  uifferent  area,  it  is  suggesteu  that  the  utility  of 
tilt*  authoring  system  could  be  substantial  ly  increased  through  provision 
of  additional  tools  for  managing  the  authoring  process.  The  approach 
envisioned  includes  capturing  relevant  parameters  of  the  development 
process  and  providing  access  to  this  information  through  summary  displays 
and  reports.  Only  a  start  has  been  made  in  tins  area.  Much  remains  taat 
can  and  should  tie  done. 

finally,  it  is  recomnendod  that  the  authoring  system's  capabilities 
for  on-line  production  of  materials  intended  for  off-line  use  be  ex¬ 
panded.  This  would  lit;  particularly  useful  if  the  additional  management 
tools  mentioned  above  were  also  made  available.  Software  development  in 
Lois  area  would  include  a  means  of  producing  scrambled  programmed  texts 
from  lesson  materials  and  author-supplied  decision  logic  and  a  text 
arcnivor.  Authors  could  develop  materials  on-line,  use  the  Cal  Presenta¬ 
tion  Program  and  its  embedded  data  collection  routines  for  formative 
evaluation,  make  needed  revisions  on-line,  print  the  number  of  copies 
needed,  and  allow  the  material  to  be  removed  to  tape.  When  additional 
copies  are  needed  or  revisions  are  required,  tno  author  could  place  an 
archive  reguest  to  move  the  lesson  material  from  tape  to  disk  for  revis¬ 
ion  or  printing.  Such  an  approach  would  not  only  facilitate  the  process 
of  development  and  formative  evaluation,  it  could  drastically  reduce 
materials  reproduction  requirements.  Currently,  it  is  common  practice 
to  reguest  many  more  copies  of  materials  titan  are  required  so  as  to  allow 
for  normal  classroom  wear  and  tear.  The  materials  are  then  often  revised 
before  many  of  the  extra  copies  are  ever  put  to  use.  The  approach 
suggested  here  would  eliminate  the  need  for  these  extra  copies. 
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